code/experimental-notebooks/6-DL-based-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "337221df",
   "metadata": {},
   "source": [
    "# Deep Learning based classification\n",
    "\n",
    "In the previous notebooks `4-CSP-based-classification` and `5-CSP-params-new-subject`, we used CSP as a way to engineer feautures from raw EEG data and used common ML classifiers to use those features for classification.\n",
    "This had very varying results.\n",
    "As said by Kostas et al ([2021](https://doi.org/10.3389/fnhum.2021.653659)): \n",
    ">Deep Neural Network (DNN) models used for classifying EEG data thus need to both develop useful features from EEG signals and subsequently classify those features.\n",
    ">This frames both the promise and the challenge of using DNNs for supervised EEG classification. \n",
    ">On the one hand, it promises to almost entirely circumvent the need for feature engineering, but on the other hand, both feature discovery and classification need to be learned from a limited supply of (relevant) high-dimensional data.\n",
    "\n",
    "\n",
    "In this notebook we will explore 3 common Deep Learnging (DL) algorithms for EEG classifications: [EEGNet V2](http://iopscience.iop.org/article/10.1088/1741-2552/aace8c/meta), [DeepConvNet](https://doi.org/10.1002/hbm.23730) and [ShallowConvNet](https://doi.org/10.1002/hbm.23730).\n",
    "It is noted that these models are designed with larger windows in mind (often 2s and more).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5341c6d",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- Checking requirements\n",
    "  - Correct anaconda environment\n",
    "  - Correct module access\n",
    "  - Correct file access\n",
    "- Loading in data\n",
    "- EEGNet\n",
    "  - Fixed window classification: Single trial | EEGNet | three class MI task | 100Hz input signal\n",
    "  - Fixed window classification: Multi trial training with unseen trial testing | EEGNet | three class MI task | 100Hz input signal\n",
    "- ShallowConvNet\n",
    "  - Fixed window classification: Single trial | ShallowConvNet | three class MI task | 100Hz input signal\n",
    "  - Fixed window classification: Multi trial training with unseen trial testing | ShallowConvNet | three class MI task | 100Hz input signal\n",
    "- DeepConvNet\n",
    "  - Fixed window classification: Single trial | DeepConvNet | three class MI task | 100Hz input signal\n",
    "  - Fixed window classification: Multi trial training with unseen trial testing | DeepConvNet | three class MI task | 100Hz input signal"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "292165d3",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## Checking requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f55ad17",
   "metadata": {},
   "source": [
    "### Correct anaconda environment\n",
    "\n",
    "The `bci-master-thesis` anaconda environment should be active to ensure proper support. Installation instructions are available on [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "334d5fd4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Active environment: bci-master-thesis\n",
      "Correct environment: True\n",
      "\n",
      "Python version: 3.8.10\n",
      "Correct Python version: True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING FOR RIGHT ANACONDA ENVIRONMENT\n",
    "####################################################\n",
    "\n",
    "import os\n",
    "from platform import python_version\n",
    "from pathlib import Path\n",
    "from copy import copy\n",
    "\n",
    "print(f\"Active environment: {os.environ['CONDA_DEFAULT_ENV']}\")\n",
    "print(f\"Correct environment: {os.environ['CONDA_DEFAULT_ENV'] == 'bci-master-thesis'}\")\n",
    "print(f\"\\nPython version: {python_version()}\")\n",
    "print(f\"Correct Python version: {python_version() == '3.8.10'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22166668",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Correct module access\n",
    "\n",
    "The following codeblock will load in all required modules and show if the versions match those that are recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ab632204",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MNE version (1.0.2 recommended): 1.0.2\n",
      "PyRieMann version (0.2.7 recommended): 0.2.7\n",
      "Numpy version (1.21.5 recommended): 1.21.5\n",
      "Pandas version (1.4.1 recommended): 1.4.1\n",
      "Scikit-learn version (1.0.2 recommended): 1.0.2\n",
      "TensorFlow version (2.8.0 recommended): 2.8.0\n",
      "Keras version (2.8.0 recommended): 2.8.0\n",
      "Pickle version (4.0 recommended): 4.0\n",
      "Matplotlib version (3.5.1 recommended): 3.5.1\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# LOADING MODULES\n",
    "####################################################\n",
    "\n",
    "# Load util function file\n",
    "import sys\n",
    "sys.path.append('../utils')\n",
    "import CLA_dataset\n",
    "\n",
    "# Load EEGModels\n",
    "import EEGModels\n",
    "from EEGModels import EEGNet, ShallowConvNet, DeepConvNet\n",
    "\n",
    "# IO functions\n",
    "from IPython.utils import io\n",
    "\n",
    "# Set logging level for MNE before loading MNE\n",
    "os.environ['MNE_LOGGING_LEVEL'] = 'WARNING'\n",
    "\n",
    "# Modules tailored for EEG data\n",
    "import mne; print(f\"MNE version (1.0.2 recommended): {mne.__version__}\")\n",
    "\n",
    "import pyriemann as prm; print(f\"PyRieMann version (0.2.7 recommended): {prm.__version__}\")\n",
    "\n",
    "# Data manipulation modules\n",
    "import numpy as np; print(f\"Numpy version (1.21.5 recommended): {np.__version__}\")\n",
    "import pandas as pd; print(f\"Pandas version (1.4.1 recommended): {pd.__version__}\")\n",
    "\n",
    "# ML libraries\n",
    "import sklearn;  print(f\"Scikit-learn version (1.0.2 recommended): {sklearn.__version__}\")\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import ConfusionMatrixDisplay, accuracy_score\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Deep Learning libraries\n",
    "import tensorflow as tf;  print(f\"TensorFlow version (2.8.0 recommended): {tf.__version__}\")\n",
    "\n",
    "import keras; print(f\"Keras version (2.8.0 recommended): {keras.__version__}\")\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "\n",
    "# Storing files\n",
    "import pickle;  print(f\"Pickle version (4.0 recommended): {pickle.format_version}\")\n",
    "\n",
    "# Plotting\n",
    "import matplotlib; print(f\"Matplotlib version (3.5.1 recommended): {matplotlib.__version__}\")\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813bb5de",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Correct file access\n",
    "\n",
    "As mentioned, this experimental notebook uses a database provided by [Kaya et al](https://doi.org/10.1038/sdata.2018.211). The CLA dataset in particular. Instructions on where to get the data are available on [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis). These instructions are under `bci-master-thesis/code/data/CLA/README.md`. FIF files from this same dataset are also made available in [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis). A check on the availability of these two datasets is performed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "caa1d182",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Full Matlab CLA file access: True\n",
      "Full MNE CLA file access: True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING FILE ACCESS\n",
    "####################################################\n",
    "\n",
    "# Use util to determine if we have access\n",
    "print(\"Full Matlab CLA file access: \" + str(CLA_dataset.check_matlab_files_availability()))\n",
    "print(\"Full MNE CLA file access: \" + str(CLA_dataset.check_mne_files_availability()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a71dde",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## Checking TensorFlow GPU support\n",
    "\n",
    "If you want to use TensorFlow with GPU acceleration, the below codeblock can help you gather insight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8d8d50f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num GPUs Available:  1\n",
      "[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\n",
      "\n",
      "\n",
      "Num CPUs Available:  1\n",
      "[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]\n"
     ]
    }
   ],
   "source": [
    "print(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))\n",
    "print(tf.config.list_physical_devices('GPU'))\n",
    "\n",
    "\n",
    "print(\"\\n\\nNum CPUs Available: \", len(tf.config.list_physical_devices('CPU')))\n",
    "print(tf.config.list_physical_devices('CPU'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1629ba2",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## Loading in data\n",
    "\n",
    "In this step, we load the data.\n",
    "The loaded data is that of a specific subject and thus can exist of multiple MNE RAW objects.\n",
    "This list of objects is provided as well as a singular one, namely the first of the list.\n",
    "\n",
    "It is important to note that the data stored by MNE is in volts meaning the values are very small.\n",
    "This is not ideal for DL methods, and thus it is recommended to multiply the data before feeding it to the DL pipeline.\n",
    "\n",
    "Remember the meaning of the markers:\n",
    "- 0: “blank” or nothing is displayed in eGUI\n",
    "    - Can be seen as a break between stimuli, thus random EEG data that should probably be ignored\n",
    "- 1: Left hand action\n",
    "    - EEG data for MI of the left hand\n",
    "- 2: Right hand action\n",
    "    - EEG data for MI of the right hand\n",
    "- 3: Passive/neutral\n",
    "    - EEG data for MI of neither left nor right hand but 'focused'\n",
    "- 91: inter-session rest break period\n",
    "- 92: experiment end\n",
    "- 99: initial relaxation period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "399f5c6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# LOADING RAW MNE DATA AND EPOCS\n",
    "####################################################\n",
    "\n",
    "def load_mne_raws(subject: str):\n",
    "    return [raw.load_data() for raw in CLA_dataset.get_raw_mne_data_for_subject(subject)]\n",
    "\n",
    "def load_mne_raw(subject: str, index: int):\n",
    "    return CLA_dataset.get_raw_mne_data_for_subject(subject)[index].load_data()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26506383",
   "metadata": {},
   "source": [
    "<hr> <hr>\n",
    "\n",
    "## Global variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "49eec473",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# Global vars\n",
    "####################################################\n",
    "\n",
    "def tensorboard_callback(log_name: str):\n",
    "    return tf.keras.callbacks.TensorBoard('./logs/' + log_name,\n",
    "                                          update_freq=\"batch\",\n",
    "                                          profile_batch=0)\n",
    "\n",
    "def lowest_loss_model_save_callback(filepath: str):\n",
    "    filepath = filepath + \"_lowest_loss_model.hdf5\"\n",
    "    return ModelCheckpoint(filepath=filepath,\n",
    "                           monitor= 'val_loss',\n",
    "                           verbose=1, \n",
    "                           save_best_only=True,\n",
    "                           mode= 'min')\n",
    "    \n",
    "def load_lowest_loss_model(filepath: str, custom_objects = {}):\n",
    "    filepath = filepath + \"_lowest_loss_model.hdf5\"\n",
    "    return keras.models.load_model(filepath, custom_objects=custom_objects)\n",
    "\n",
    "def highest_accuracy_model_save_callback(filepath: str):\n",
    "    filepath = filepath + \"_highest_acc_model.hdf5\"\n",
    "    return ModelCheckpoint(filepath=filepath,\n",
    "                           monitor= 'val_accuracy',\n",
    "                           verbose=1, \n",
    "                           save_best_only=True,\n",
    "                           mode= 'max')\n",
    "    \n",
    "def load_highest_accuracy_model(filepath: str, custom_objects = {}):\n",
    "    filepath = filepath + \"_highest_acc_model.hdf5\"\n",
    "    return keras.models.load_model(filepath, custom_objects=custom_objects)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1960330e",
   "metadata": {},
   "source": [
    "<hr> <hr>\n",
    "\n",
    "## Tensorboard\n",
    "\n",
    "To launch the tensorboard use the following command in the `experimental-notebooks` folder:\n",
    "- Windows: `tensorboard --logdir=./logs/`\n",
    "- MacOS: `tensorboard --logdir='./logs/'`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b35576",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## EEGNet\n",
    "\n",
    "EEGNet is a compact convolutional neural network for EEG-based brain–computer interfaces by Lawhern et al ([2018](https://iopscience.iop.org/article/10.1088/1741-2552/aace8c/meta)). There were two proposed version, the latter of which is the referenced published version. Since the latter performs fat better and is the most commonly known EEGNet, this V2 is used here. The EEGModels library provided by the Army Research Laboratory on [GitHub](https://github.com/vlawhern/arl-eegmodels) is used for easy use of this model.\n",
    "\n",
    "### Fixed window classification: Single trial | EEGNet | three class MI task | 100Hz input signal\n",
    "\n",
    "The model recommends using 128 samples.\n",
    "If you remember from the previous notebooks, the data was sampled at 200Hz, which means there are 200 samples per second.\n",
    "In the previous notebook we processed data of 0.5 seconds, thus 100 samples.\n",
    "To get initial results, we will use the model on this 100 samples data.\n",
    "\n",
    "The scores are given for the best CSP approach as well as for the trained DL approach.\n",
    "Scores for the DL approach are a combination of the best model based on validation loss / validation accuracy.\n",
    "TensorBoard was used to monitor the behaviour of the model over time, to ensure no overfitting tendencies occur and that enough epochs have happened to learn optimally.\n",
    "\n",
    "**The result on subject `C` are:**\n",
    "\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** |\n",
    "|----------------|------------------------------------|---------------------------------|\n",
    "| 0              | 0.896                              | 0.944 / 0.934                   |\n",
    "| 1              | 0.847                              | 0.931 / 0.913                   |\n",
    "| 2              | 0.719                              | 0.892 / 0.878                   |\n",
    "\n",
    "**The result on subject `B` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** |\n",
    "|----------------|------------------------------------|---------------------------------|\n",
    "| 0              | 0.465                              | 0.569 / 0.587                   |\n",
    "| 1              | 0.560                              | 0.701 / 0.684                   |\n",
    "| 2              | 0.653                              | 0.816 / 0.792                   |\n",
    "\n",
    "\n",
    "**The result on subject `E` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** |\n",
    "|----------------|------------------------------------|---------------------------------|\n",
    "| 0              | 0.701                              | 0.792 / 0.785                   |\n",
    "| 1              | 0.580                              | 0.795 / 0.799                   |\n",
    "| 2              | 0.805                              | 0.864 / 0.864                   |\n",
    "\n",
    "It is important to note the training behaviour of the models has greatly improved when multiplying the data by a million.\n",
    "See the comparison of the accuracy score on the validation set for subject `0` with the file at index 0 for example.\n",
    "All settings of the model are equal but one has multiplied the data by a million before training.\n",
    "The better behaving model is that with the data multiplied.\n",
    "\n",
    "![Comparison of models using multiplied data vs non multiplied data](figures/6-dl/eegnet_multiplied_data.PNG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "14156d3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 0 ... 666799  =      0.000 ...  3333.995 secs...\n",
      "Using data from preloaded Raw for 960 events and 801 original time points ...\n",
      "0 bad epochs dropped\n",
      "Loaded fixed window binary epochs:\n",
      "\n",
      "Extracted labels from epochs: [1 2 1 3 1 3 3 2 3 2]\n",
      "One Hot Encoded labels: [[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]]\n",
      "Labels match before and after the One Hot Encoding: True\n",
      "Shape of data (epochs, channels, samples): (960, 21, 100)\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# PREPPING THE DATA\n",
    "####################################################\n",
    "\n",
    "# Specify raw to use\n",
    "mne_raw = load_mne_raw(subject= \"C\", index=0)\n",
    "\n",
    "# Get the epoch from the RAW limited to MI tasks\n",
    "# Include period before and after to enable filtering possibilities\n",
    "mne_fixed_window_epochs = CLA_dataset.get_usefull_epochs_from_raw(mne_raw, start_offset=-1.5, end_offset=1.5)['task/neutral', 'task/left', 'task/right']\n",
    "\n",
    "# Load the epochs, we don't need filtering since this is what we want to learn!\n",
    "mne_fixed_window_epochs.load_data()\n",
    "print(f\"Loaded fixed window binary epochs:\\n\")\n",
    "\n",
    "# Labels: should be one hot encoded!\n",
    "labels = mne_fixed_window_epochs.events[:, -1]\n",
    "print(f\"Extracted labels from epochs: {labels[:10]}\")\n",
    "\n",
    "# Go to 2D representation\n",
    "labels = labels.reshape(-1, 1)\n",
    "\n",
    "# One Hot Encode the labels\n",
    "ohe = OneHotEncoder()\n",
    "ohe_labels = ohe.fit_transform(labels).toarray()\n",
    "print(f\"One Hot Encoded labels: {ohe_labels[:10]}\")\n",
    "\n",
    "# Show ohe labels\n",
    "np.shape(ohe_labels)\n",
    "ohe_labels[:10]\n",
    "\n",
    "# Validate OHE\n",
    "print(f\"Labels match before and after the One Hot Encoding: {np.array_equal(ohe.inverse_transform(ohe_labels), labels)}\")\n",
    "\n",
    "# Get effective data (half a second)\n",
    "mne_fixed_window_epochs_data = mne_fixed_window_epochs.get_data(tmin=0.2, tmax=0.7)\n",
    "\n",
    "# Fix scaling sensitivity as MNE stores as data * 10e-6\n",
    "mne_fixed_window_epochs_data = mne_fixed_window_epochs_data * 1000000\n",
    "\n",
    "# Delete unused variables\n",
    "del mne_fixed_window_epochs\n",
    "print(f\"Shape of data (epochs, channels, samples): {np.shape(mne_fixed_window_epochs_data)}\")\n",
    "\n",
    "# Remove unused variables\n",
    "del labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ca901652",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 21, 100, 1)]      0         \n",
      "                                                                 \n",
      " conv2d (Conv2D)             (None, 21, 100, 8)        400       \n",
      "                                                                 \n",
      " batch_normalization (BatchN  (None, 21, 100, 8)       32        \n",
      " ormalization)                                                   \n",
      "                                                                 \n",
      " depthwise_conv2d (Depthwise  (None, 1, 100, 16)       336       \n",
      " Conv2D)                                                         \n",
      "                                                                 \n",
      " batch_normalization_1 (Batc  (None, 1, 100, 16)       64        \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation (Activation)     (None, 1, 100, 16)        0         \n",
      "                                                                 \n",
      " average_pooling2d (AverageP  (None, 1, 25, 16)        0         \n",
      " ooling2D)                                                       \n",
      "                                                                 \n",
      " spatial_dropout2d (SpatialD  (None, 1, 25, 16)        0         \n",
      " ropout2D)                                                       \n",
      "                                                                 \n",
      " separable_conv2d (Separable  (None, 1, 25, 16)        512       \n",
      " Conv2D)                                                         \n",
      "                                                                 \n",
      " batch_normalization_2 (Batc  (None, 1, 25, 16)        64        \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_1 (Activation)   (None, 1, 25, 16)         0         \n",
      "                                                                 \n",
      " average_pooling2d_1 (Averag  (None, 1, 3, 16)         0         \n",
      " ePooling2D)                                                     \n",
      "                                                                 \n",
      " spatial_dropout2d_1 (Spatia  (None, 1, 3, 16)         0         \n",
      " lDropout2D)                                                     \n",
      "                                                                 \n",
      " flatten (Flatten)           (None, 48)                0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 3)                 147       \n",
      "                                                                 \n",
      " softmax (Activation)        (None, 3)                 0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,555\n",
      "Trainable params: 1,475\n",
      "Non-trainable params: 80\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE EEGNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Create the TensorFlow Keras model\n",
    "keras_eegnet_model = EEGNet(\n",
    "    nb_classes = 3, # int, number of classes to classify. \n",
    "    Chans = 21, # number of channels in the EEG data. \n",
    "    Samples = 100, # number of time points in the EEG data. (default: 128)\n",
    "    dropoutRate = 0.5, # dropout fraction. (default: 0.5)\n",
    "    kernLength = 50, # length of temporal convolution in first layer. Suggested: half the sampling rate. (default: 64)\n",
    "    F1 = 8, # number of temporal filters. (default: 8)\n",
    "    F2 = 16, # number of pointwise filters. (default: 16)\n",
    "    D = 2, # number of spatial filters to learn within each temporal convolution. (default: 2)\n",
    "    norm_rate = 0.25, # Normalisation rate. (default: 0.25)\n",
    "    dropoutType = 'SpatialDropout2D' # Either SpatialDropout2D or Dropout, passed as a string. (default: Dropout)\n",
    "    )\n",
    "\n",
    "# Compile the model so it can be fitted\n",
    "# Loss and optimizer from EEGNet paper\n",
    "keras_eegnet_model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics=[\"accuracy\"])\n",
    "\n",
    "# Show summary of the model\n",
    "keras_eegnet_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2f898be9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for lowest loss model\n",
      "Trained EEGNet on single session using train/test split and got accuracy of: 0.9444444444444444\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbkUlEQVR4nO3deZhV1Znv8e+vBuZ5EJAh4oRBW9EmRuONcUoknaQxPulEzTXetIkZNA7p3LTptq9p06bjNdFMxohDazrEIYrBOKEoXqOJCCIigwgBRQRkFpCpqHrvH2dXLAzUOQfOqb138fs8z344e+9z1n45FG+ttfZaaysiMDPLs5q0AzAz21tOZGaWe05kZpZ7TmRmlntOZGaWe3VpB9BSvz61ccDQ+rTDyKxXZ3VJO4Tsk9KOINO2xjtsj6179SWdfnLXWLO2saT3vjBr26SIGLM31ytFphLZAUPreX7S0LTDyKzTh/xt2iFknuoz9SOdOc9te2Svy1i9tpGpk4aU9N76QX/ut9cXLIH/1c2sTEFjNKUdxE6cyMysLAE0ka2B9E5kZla2JlwjM7McC4IGNy3NLM8CaHTT0szyzn1kZpZrATRmbNUcJzIzK1u2esicyMysTEG4j8zM8i0CGrKVx5zIzKxcopFszWl1IjOzsgTQ5BqZmeWda2RmlmuFAbFOZGaWYwE0RLbWZHUiM7OyBKIxY4tLO5GZWdmawk1LM8sx95GZWTsgGt1HZmZ5Vlgh1onMzHIsQmyP2rTD2IkTmZmVrcl9ZGaWZ4XOfjctzSzX3NlvZjnnzn4zaxcaPSDWzPIsEA2RrdSRrWjMLPOy2NmfrWjMLPMC0RilbcVIukzSHEmzJd0pqZOk4ZKmSloo6W5JHYqV40RmZmVroqakrTWSBgMXA6Mj4gigFjgLuAa4PiIOBtYB5xeLx4kscf8t/bjg5BF8+aQRTLi5/07n7v1lf07ffxRvr8nWaOa0fPOHr3P3zFncNHlu2qFkUr9B27jmN/O46bFZ3DTpZcb+rxVph1RREdAYNSVtJagDOkuqA7oAy4FTgHuT83cAZxQrpGqJTNJtklZKml2ta1TKa6904pHxffnpQ6/yy8nzmfp4D95cXKjNrnyznhn/rzv7Dd6ecpTZ8dhv+/Cv//PgtMPIrKYd4uarh/GVjx3JpWeO5FNfeIthB29JO6yKKXT215a0Af0kTW+xXfCXciLeBH4ILKGQwN4GXgDWR8SO5G1LgcHFYqpmjex2YEwVy6+YJQs6ctjRm+nUJaitgyOP38SzD/cC4KbvDub8K5ahbN1tTtXsqd3ZuN61091Zu6oDC+d0BWDLO7W8sbAzfQe2r1+EjdSUtAGrI2J0i21ccxmSegNjgeHA/kBX9jBnVC2RRcTTwNpqlV9JBxy2ldnPd2XD2lq2bhbTnuzBqmX1/PHRHvQb2MBBh29NO0TLqQGDt3HQyM3Mn9kt7VAqJhBNUdpWxGnA4ohYFRENwATgBKBX0tQEGAK8WawgD78Ahh2yjc9+fSXfOfsgOnVp4sDDt9CwXdz1swH8551/Tjs8y6lOXRq54sYF3PS9YWze1L5qsBUafrEEOE5SF2ALcCowHZgCfAa4CzgPmFisoNQ7+yVd0Nx+XrWmMbU4xpyzlhsmvcqP7l9It56NvG/EVlYs6cDXTjuMLxw7klXL67nw9BGsXencb8XV1jXxbzcuYMrEvjw7qU/a4VRU4bmWNSVtrZYTMZVCp/4M4GUK+Wgc8M/ANyUtBPoCtxaLKfX/lUmbeRzA6KM6pfbYz/Wr6+jVbwcrl9bz7MM9+cmDC/j0l1b/5fwXjh3Jzx6ZT8++6SVby4vgsmsWs2RhZybcOijtYKqgck8aj4grgSvfc3gRcGw55aSeyLLiqi8dwMZ1ddTWBxd9fyndejph7c7lP1/MkcdvpGefHfx62sv8948GMemufmmHlRmHj97EaWeuYfErnbnhocJN+9uvHcK0p3qlG1iFFB4Hl62mctUSmaQ7gZMo3H5dClwZEUWriGm57ncLWz3/q+c9ZqrZDy4annYImTZnenfGDC+rQpErESrabGxrVUtkEXF2tco2s3R5PTIzy7XCemTZGljpRGZmZfIKsWaWc4XhF66RmVmONc+1zBInMjMrm9fsN7NcKyzj46almeWc+8jMLNcKq1+4aWlmOVaYouREZma55hqZmbUDHtlvZrnmu5Zm1i64aWlmuda8Zn+WOJGZWVkC2OEamZnlnZuWZpZvpT3qrU05kZlZWbywopm1C66RmVmueWFFM8u9QOxocme/meWc+8jMLN/CTUszyzn3kZlZu+BEZma5FohGd/abWd65s9/Mci3c2W9m7UE4kZlZvnnSuJm1A66RteLVWV04ff9RaYeRWfctfTbtEDLvMwd9JO0Q2r0IaGxyIjOznPNdSzPLtSB7TctsjWozsxwodPaXshUtSeol6V5Jr0iaJ+l4SX0kPS5pQfJn72LlOJGZWdkiSttK8BPg0Yg4DDgKmAdcDjwREYcATyT7rXIiM7OyRaikrTWSegInArcWyoztEbEeGAvckbztDuCMYvG4j8zMylK4a1lyHaifpOkt9sdFxLjk9XBgFfBfko4CXgAuAQZExPLkPSuAAcUu4kRmZmUrsdkIsDoiRu/mXB1wDPCNiJgq6Se8pxkZESGp6NXctDSzslWiaQksBZZGxNRk/14Kie0tSYMAkj9XFivIiczMyhKUlsSKJbKIWAG8IWlEcuhUYC7wAHBecuw8YGKxmNy0NLOyld6yLOobwHhJHYBFwBcpVLDukXQ+8Drw2WKFOJGZWXkCokJTlCJiJrCrPrRTyynHiczMypa1kf1OZGZWtjLuWraJ3SYyST+jlaZwRFxclYjMLNOyONeytRrZ9FbOmdm+KoC8JLKIuKPlvqQuEbG5+iGZWdZlrWlZdBxZMht9LvBKsn+UpF9UPTIzyygRTaVtbaWUAbE/Bk4H1gBExEsUJnqa2b4qStzaSEl3LSPiDWmn7NpYnXDMLPMiX539zd6Q9CEgJNVTmJ0+r7phmVmm5a2PDPgqcCEwGFgGjEr2zWyfpRK3tlG0RhYRq4HPt0EsZpYXTWkHsLNS7loeKOn3klZJWilpoqQD2yI4M8ug5nFkpWxtpJSm5W+Ae4BBwP7Ab4E7qxmUmWVbBdfsr4hSElmXiPjviNiRbL8GOlU7MDPLsLwMv5DUJ3n5iKTLgbsohPY54OE2iM3MsipHwy9eoJC4miP+SotzAXynWkGZWbYVX0W/bbU213J4WwZiZjkRgjacflSKkkb2SzoCGEmLvrGI+FW1gjKzjMtLjayZpCuBkygksoeBjwPPAE5kZvuqjCWyUu5afobC+tkrIuKLFB5r3rOqUZlZtuXlrmULWyKiSdIOST0oPGNuaJXjStXokzbw1e8to7YmeOTOPtzz86IPOt4nPHjLQCbfuR8R8NFzVvLJL63gju8NY/rk3tTVNzHwfdu46Lo/07Xnvr2mQL9B2/jfP1pEr34NEOLhO/sz8faBaYdVORlcWLGUGtl0Sb2AmyncyZwB/KnYhyQNlTRF0lxJcyRdsnehto2amuDC77/JFZ8fzpdPGsHJY9cz7JCtaYeVuiWvdGbynftxzYOzue6xWUyf3Jvlizty1Ilv8+MnXuL6yS+z/4FbmfDzwWmHmrqmHeLmq4fxlY8dyaVnjuRTX3iLYQdvSTusilKUtrWVooksIr4eEesj4pfAR4HzkiZmMTuAf4qIkcBxwIWSRu5duNU34ujNLHutAyuWdGRHQw1PTezF8ae/nXZYqVu6sDOHjNpEx85N1NbB4cdtYOojfRj1kbepTer1hx6zkTXLO6QbaAasXdWBhXO6ArDlnVreWNiZvgO3pxxVhWWsabnbRCbpmPduQB+gLnndqohYHhEzktcbKSz9k/lf130HNrBq2bv/GVcvr6ffoIYUI8qGYSM2M+/57mxcV8e2LTXMeLIXq5d13Ok9T9y9H0efvD6dADNqwOBtHDRyM/Nndks7lIrKWo2stT6yH7VyLoBTSr2IpAOAo4Gpuzh3AXABQCe6lFqktbEhh2zljK8v46pzDqNjlyYOOHwzNbXv/qTe+9P9qa0NTjxzdYpRZkunLo1cceMCbvreMDZvqk07nMrKWB9ZawNiT67EBSR1A+4DLo2IDbu4zjhgHEAP9Un9pu6aFfX03//dZkC/QQ2sXl6fYkTZcdrZqzjt7FUAjP/BUPoOKnxPT97Tnxcm9+a7d89D2fr5Tk1tXRP/duMCpkzsy7OT+hT/QJ60cbOxFKV09u+xZEXZ+4DxETGhmteqlPkzuzB4+HYGDN1GXX0TJ41dz3OPebQJwNurC7/3Vr3Zgece6cOHz1jNi1N6MvHGQVz+X/Pp2Dlji1SlJrjsmsUsWdiZCbcOSjuY6shYH1nVnjSuwiL/twLzIuK6al2n0poaxQ3/Opjv/2YRNbXw2F19eP1VL/YBcO0Fh7JxXR21dcGXr15M156N3HLFcBq2i6vOfj8Ahx6zia/8YHHKkabr8NGbOO3MNSx+pTM3PDQbgNuvHcK0p3qlG1gFKWO/s6qWyIATgHOBlyXNTI79S0RkfuWMaU/2YNqTPdIOI3P+Y8Lcvzp2w7Mz2z6QjJszvTtjhh+bdhjVlbGmZSlTlERhqesDI+IqScOAgRHxfGufi4hnaMtFu82sTbT1HclSlNJH9gvgeODsZH8jcEPVIjKz7MvYUtelNC0/GBHHSHoRICLWSfKoR7N9WcZqZKUksgZJtSShS+pP5p6hYmZtKWtNy1IS2U+B+4H9JF1NYTWMK6oalZllV+TwrmVEjJf0AoWlfAScERF+0rjZvixvNbLkLuVm4Pctj0XEkmoGZmYZlrdEBjzEuw8h6QQMB+YDh1cxLjPLsKz1kZWyjM/fRMSRyZ+HAMdSwnpkZmalkFQr6UVJDyb7wyVNlbRQ0t2ljJIoe65lsjTPB/cgXjNrLyo71/ISCst8NbsGuD4iDgbWAecXK6CUPrJvttitAY4BlpUcopm1LxW8aylpCPAJ4Grgm8lMolOAc5K33AF8F7ixtXJK6SPr3uL1Dgp9ZveVGa+ZtSel17b6SZreYn9csnRXsx8D3+bdPNMXWB8RO5L9pZSwIGuriSwZCNs9Ir5VatRm1r6Jsjr7V0fE6F2WI30SWBkRL0g6aW9i2m0ik1QXETsknbA3FzCzdqgydy1PAP5e0t9RGBHRA/gJ0Ks5/wBDgDeLFdRaZ3/z6hYzJT0g6VxJZzZve/kXMLO8KnG9/mK1toj4TkQMiYgDgLOAJyPi88AUCjOIAM4DJhYLqZQ+sk7AGgodcM3jyQLIxYqvZlYF1Z2i9M/AXZL+A3iRwgKtrWotke2X3LGczbsJrFnGhsOZWVuq9IDYiHgKeCp5vYjCeNWStZbIaoFu7HpxRCcys31ZxjJAa4lseURc1WaRmFk+ZPApSq0lMi9TbWa7lLW5lq0lslPbLAozy5e8JLKIWNuWgZhZfuRuYUUzs53krI/MzOyviOx1oDuRmVn5XCMzs7zL011LM7NdcyIzs1zL4+PgzMz+imtkZpZ37iMzs/xzIts9dainbuCQtMPIrH8Y4VljxVz1ylNph5BpX/zUpoqU4xqZmeVbUO2FFcvmRGZmZSnz4SNtwonMzMrnRGZmeafIViZzIjOz8nj1CzNrD9xHZma55ylKZpZ/rpGZWa6V8BTxtuZEZmblcyIzszzzgFgzaxfUlK1M5kRmZuXxODIzaw88/MLM8s81MjPLO3f2m1m+BeBJ42aWd+4jM7Nc8zgyM8u/CDctzSz/XCMzs/zLWCKrSTsAM8sfRWlbq2VIQyVNkTRX0hxJlyTH+0h6XNKC5M/exeJxIjOz8gTQGKVtrdsB/FNEjASOAy6UNBK4HHgiIg4Bnkj2W+VEZmZlq0SNLCKWR8SM5PVGYB4wGBgL3JG87Q7gjGLxuI/MzMpX4buWkg4AjgamAgMiYnlyagUwoNjnncjMrGxl3LXsJ2l6i/1xETFup7KkbsB9wKURsUHSX85FREjFr+ZEZmblKW8Zn9URMXp3JyXVU0hi4yNiQnL4LUmDImK5pEHAymIXcR+ZmZVFgBqjpK3VcgpVr1uBeRFxXYtTDwDnJa/PAyYWi8k1MjMrW4WeNH4CcC7wsqSZybF/AX4A3CPpfOB14LPFCnIiM7PyVGiF2Ih4hkIFb1dOLacsJ7JdOOOsRXxs7BtEwOt/7sH13zuShu21aYeVGfUdmrj2N7Op79BEbV3wzKN9+fVPh6UdVur+eOsApt/dHykYMGILn752MXUdgsk/HMych/ug2uDYz6/k+C8W7fLJuH1orqWkTsDTQMfkOvdGxJXVul6l9O2/lU997jW+dtZH2L6tlsuvnsFHPrqMyQ8NTTu0zGjYLi7/wuFs3VxLbV0TP7xrNtOf7s0rM7unHVpqNqyo50+3D+Dix1+mvlNw14UH8fLv+0CIt5d34OInXqamBjatbh91h6zNtaxmZ/824JSIOAoYBYyRdFwVr1cxtbVBh46N1NQ20bFTI2tWd0o7pIwRWzcXaqh1dUFdXWTtF3QqmhpFw9YaGndAw9YaeuzXwPO/7s/JFy+jJvmf1q3fjnSDrJTmFTCKbW2kar8eIiKATclufbJl/sd9zapOTBh/ILdPfJLt22qZMbUfL07tn3ZYmVNTE/z0dy+x/7CtPDh+IPNf2ndrYwA9BjbwP768gh+dcBR1nZo4+MMbOPjEDdxzyUG8/GAf5j3Wm659GvjElUvoO3xb2uHunaDoHcm2VtXhF5Jqk7sRK4HHI2JqNa9XCd26N3DciW/xj58+mXM/cSqdOjdy8pilaYeVOU1N4qK/H8W5Hx7NoUdu4n2HvJN2SKna8nYt8x7vxTefnsW3n3uJ7ZtrmHl/Xxq3i/qOTXztgbn87Vmruf/bw9MOtTKixK2NVDWRRURjRIwChgDHSjrive+RdIGk6ZKmb2/cUs1wSjLqA6t5a1lnNqzvSGNjDX+cMpD3/826tMPKrHc21jFrak9Gn7g+7VBS9ednetB76Da69t1BbX0w8vR1vDGjGz0GbmfkmMLPz8jT17FifueUI60MRZS0tZU2GRAbEeuBKcCYXZwbFxGjI2J0h9r0/5FXvdWJEUesp2PHRiA46gOreeO1bmmHlSk9+zTQtXuhr6dDx0aO/tB63liU/r9dmnruv503XuzG9i01RMCiP/ag/0FbeP/H1rPoTz0AeG1qd/rlvVnZbF/pI5PUH2iIiPWSOgMfBa6p1vUqZf6c3jz75CB+8qs/0NgoFr3ak0d+56EFLfXuv51v/d+F1NQEqgn+8Eg/np/SJ+2wUjX06Hc4/ONrufGTI6mpCwaN3Mzos1fRsK2Gey89kD/eNoCOXZoY+5+L0w517wWwDz18ZBBwh6RaCjW/eyLiwSper2LG33wo428+NO0wMuu1+V25aOxRaYeROadetoxTL1u207G6jo2ce9uClCKqDtG2zcZSVPOu5SwKy3KYWXvTlK0qWfsYnWdmbWcfa1qaWTu1zzQtzawdcyIzs3zbhyaNm1k71fwUpQxxIjOzsrmPzMzyz4nMzHItgCYnMjPLNXf2m1l74ERmZrkWQGO2hvY7kZlZmQLCiczM8s5NSzPLNd+1NLN2wTUyM8s9JzIzy7UIaGxMO4qdOJGZWflcIzOz3HMiM7N8C9+1NLOcCwgPiDWz3PMUJTPLtQg/Ds7M2gF39ptZ3oVrZGaWb15Y0czyzpPGzSzvAoiMTVGqSTsAM8uZSBZWLGUrQtIYSfMlLZR0+Z6G5BqZmZUtKtC0lFQL3AB8FFgKTJP0QETMLbcs18jMrHyVqZEdCyyMiEURsR24Cxi7J+EoMnT3QdIq4PW042ihH7A67SAyzN9PcVn7jt4XEf33pgBJj1L4e5WiE7C1xf64iBiXlPMZYExEfCnZPxf4YERcVG5MmWpa7u0XXGmSpkfE6LTjyCp/P8W1x+8oIsakHcN7uWlpZml5ExjaYn9IcqxsTmRmlpZpwCGShkvqAJwFPLAnBWWqaZlB49IOIOP8/RTn72g3ImKHpIuASUAtcFtEzNmTsjLV2W9mtifctDSz3HMiM7PccyLbBUm3SVopaXbasWSRpKGSpkiaK2mOpEvSjilLJHWS9Lykl5Lv59/Tjqm9cx/ZLkg6EdgE/Coijkg7nqyRNAgYFBEzJHUHXgDO2JOpJe2RJAFdI2KTpHrgGeCSiHgu5dDaLdfIdiEingbWph1HVkXE8oiYkbzeCMwDBqcbVXZEwaZktz7ZXGOoIicy2yuSDgCOBqamHEqmSKqVNBNYCTweEf5+qsiJzPaYpG7AfcClEbEh7XiyJCIaI2IUhdHqx0pyF0UVOZHZHkn6fu4DxkfEhLTjyaqIWA9MATI3P7E9cSKzsiWd2bcC8yLiurTjyRpJ/SX1Sl53prDe1iupBtXOOZHtgqQ7gT8BIyQtlXR+2jFlzAnAucApkmYm29+lHVSGDAKmSJpFYT7h4xHxYMoxtWsefmFmuecamZnlnhOZmeWeE5mZ5Z4TmZnlnhOZmeWeE1mOSGpMhjrMlvRbSV32oqzbk6fYIOkWSSNbee9Jkj60B9d4TdJfPW1nd8ff855NrZ3fxfu/K+lb5cZo7YMTWb5siYhRyYoc24GvtjwpaY+WLo+ILxVZueIkoOxEZtZWnMjy6w/AwUlt6Q+SHgDmJpOVr5U0TdIsSV+Bwmh8ST9PHk8/GdivuSBJT0kanbweI2lGspbWE8mk8K8ClyW1wQ8nI9fvS64xTdIJyWf7SnosWYPrFkDF/hKSfifpheQzF7zn3PXJ8Sck9U+OHSTp0eQzf5B0WEW+Tcs1P3wkh5Ka18eBR5NDxwBHRMTiJBm8HREfkNQReFbSYxRWqBgBjAQGAHOB295Tbn/gZuDEpKw+EbFW0i+BTRHxw+R9vwGuj4hnJA2j8PCI9wNXAs9ExFWSPgGUMiPiH5NrdAamSbovItYAXYHpEXGZpP+TlH0RhYd5fDUiFkj6IPAL4JQ9+BqtHXEiy5fOydIwUKiR3Uqhyfd8RCxOjn8MOLK5/wvoCRwCnAjcGRGNwDJJT+6i/OOAp5vLiojdrcl2GjCyMOUSgB7JShgnAmcmn31I0roS/k4XS/p08npoEusaoAm4Ozn+a2BCco0PAb9tce2OJVzD2jknsnzZkiwN8xfJf+h3Wh4CvhERk97zvkrOhawBjouIrbuIpWSSTqKQFI+PiM2SngI67ebtkVx3/Xu/AzP3kbU/k4CvJcvsIOlQSV2Bp4HPJX1og4CTd/HZ54ATJQ1PPtsnOb4R6N7ifY8B32jekTQqefk0cE5y7ONA7yKx9gTWJUnsMAo1wmY1QHOt8hwKTdYNwGJJ/5BcQ5KOKnIN2wc4kbU/t1Do/5qhwsNTbqJQ874fWJCc+xWF1T12EhGrgAsoNONe4t2m3e+BTzd39gMXA6OTmwlzeffu6b9TSIRzKDQxlxSJ9VGgTtI84AcUEmmzdygsSDibQh/YVcnxzwPnJ/HNAcaW8J1YO+fVL8ws91wjM7PccyIzs9xzIjOz3HMiM7PccyIzs9xzIjOz3HMiM7Pc+/8NSqn7Yit9OwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Results for highest accuracy model\n",
      "Trained EEGNet on single session using train/test split and got accuracy of: 0.9340277777777778\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbM0lEQVR4nO3de5hcVZnv8e+vL0nnHnIhCUmQoJgQEQIHQhAHAohEHCfo4SjqAEdhAAFF1KPoeAZGR4+MiMMoKkEYcESQm4IiJFyCCGquhEtIgHALuScdcjO3Tvd7/qjdGGLSVZVU9d678vs8z3669t5Va79d0G/WWnuttRURmJnlWV3aAZiZ7SknMjPLPScyM8s9JzIzyz0nMjPLvYa0A9jegH71ccDwxrTDyKwXnu6edgjZJ6UdQaZtjr+wNTbv0Zd0ygk9onl1a0nvnfX0lskRMWFPrleKTCWyA4Y3Mn3y8LTDyKxThv2PtEPIPDVm6n/pzPnzlvv3uIxVq1uZNnlYSe9tHPLSgD2+YAn8X93MyhS0RlvaQbyFE5mZlSWANrI1kN6JzMzK1oZrZGaWY0HQ4qalmeVZAK1uWppZ3rmPzMxyLYDWjK2a40RmZmXLVg+ZE5mZlSkI95GZWb5FQEu28pgTmZmVS7SSrTmtTmRmVpYA2lwjM7O8c43MzHKtMCDWiczMciyAlsjWmqxOZGZWlkC0ZmxxaScyMytbW7hpaWY55j4yM6sBotV9ZGaWZ4UVYp3IzCzHIsTWqE87jLdwIjOzsrW5j8zM8qzQ2e+mpZnlmjv7zSzn3NlvZjWh1QNizSzPAtES2Uod2YrGzDIvi5392YrGzDIvEK1R2laMpEslzZX0rKRbJTVJGiFpmqQFkn4pqUuxcpzIzKxsbdSVtHVE0lDgc8CREXEIUA+cAVwJfD8i3gG8AZxTLB4nssSvfjqA804YyT+NH8nd1w98y7k7fzKQU/Ybw9rmbI1mTssXrnqNX855museei7tUDJpwJAtXPmLeVw35Wmum/wME//3srRDqqgIaI26krYSNADdJDUA3YGlwInAncn5m4HTihVStUQm6UZJKyQ9W61rVMqr85u4/5b+/Od9L/CTh55n2oO9WfxKoTa7YnEjs3/fi32Hbk05yuyYckc//vkf35F2GJnVtk1c/639Of/9h/L5j4zmQ2ctZ/93bEo7rIopdPbXl7QBAyTN3G47781yIhYDVwELKSSwtcAsYE1EbEvetggYWiymatbIbgImVLH8iln4YldGHb6Rpu5BfQMceswGnvhdXwCuu2Io53x9CcrW3eZUPTutF+vXuHa6K6tXdmHB3B4AbPpLPa8v6Eb/wbX1D2ErdSVtwKqIOHK7bVJ7GZL2ASYCI4D9gB7sZs6oWiKLiMeA1dUqv5IOGLWZZ6f3YN3qejZvFDMe6c3KJY388YHeDBjcwtvftTntEC2nBg3dwttHb+T5OT3TDqViAtEWpW1FvA94JSJWRkQLcDdwLNA3aWoCDAMWFyvIwy+A/Q/awkcvXMFXP/52mrq3ceC7NtGyVdz2g0H8v1tfSjs8y6mm7q18/ccvct0392fjhtqqwVZo+MVCYJyk7sAm4CRgJjAVOB24DTgbuKdYQal39ks6r739vLK5NbU4JnxiNddOfoHv/WoBPfu08raRm1m2sAufed8ozho7mpVLG7nolJGsXuHcb8XVN7Txf3/8IlPv6c8Tk/ulHU5FFZ5rWVfS1mE5EdModOrPBp6hkI8mAV8BviBpAdAfuKFYTKn/VSZt5kkARx7WlNpjP9esaqDvgG2sWNTIE7/rwzW/fZEPn7vqzfNnjR3ND+5/nj7900u2lhfBpVe+wsIF3bj7hiFpB1MFlXvSeERcDly+w+GXgbHllJN6IsuKb5x7AOvfaKC+Mbj424vo2ccJa1cu++ErHHrMevr028bPZzzDf39vCJNvG5B2WJnxriM38L6PNPPK/G5ce1/hpv1N3x3GjEf7phtYhRQeB5etpnLVEpmkW4HxFG6/LgIuj4iiVcS0XP3rBR2e/9l0j5lq952LR6QdQqbNndmLCSPKqlDkSoSKNhs7W9USWUR8vFplm1m6vB6ZmeVaYT2ybA2sdCIzszJ5hVgzy7nC8AvXyMwsx9rnWmaJE5mZlc1r9ptZrhWW8XHT0sxyzn1kZpZrhdUv3LQ0sxwrTFFyIjOzXHONzMxqgEf2m1mu+a6lmdUENy3NLNfa1+zPEicyMytLANtcIzOzvHPT0szyrbRHvXUqJzIzK4sXVjSzmuAamZnlmhdWNLPcC8S2Nnf2m1nOuY/MzPIt3LQ0s5xzH5mZ1QQnMjPLtUC0urPfzPLOnf1mlmvhzn4zqwXhRGZm+eZJ42ZWA1wj68ALT3fnlP3GpB1GZt208Pdph5B55xxyatohZJpa9jwBRUBrmxOZmeWc71qaWa4F2WtaZmtUm5nlQKGzv5StaElSX0l3SpovaZ6kYyT1k/SgpBeTn/sUK8eJzMzKFlHaVoJrgAciYhRwGDAPuAx4OCIOAh5O9jvkRGZmZYtQSVtHJPUBjgNuKJQZWyNiDTARuDl5283AacXicR+ZmZWlcNey5DrQAEkzt9ufFBGTktcjgJXAf0k6DJgFXAIMioilyXuWAYOKXcSJzMzKVmKzEWBVRBy5i3MNwBHAZyNimqRr2KEZGREhqejV3LQ0s7JVomkJLAIWRcS0ZP9OColtuaQhAMnPFcUKciIzs7IEpSWxYoksIpYBr0samRw6CXgOuBc4Ozl2NnBPsZjctDSzspXesizqs8AtkroALwOfolDBul3SOcBrwEeLFeJEZmblCYgKTVGKiDnAzvrQTiqnHCcyMytb1kb2O5GZWdnKuGvZKXaZyCT9gA6awhHxuapEZGaZlsW5lh3VyGZ2cM7M9lYB5CWRRcTN2+9L6h4RG6sfkpllXdaalkXHkSWz0Z8D5if7h0n6UdUjM7OMEtFW2tZZShkQ+x/AKUAzQEQ8RWGip5ntraLErZOUdNcyIl6X3pJdW6sTjpllXuSrs7/d65LeA4SkRgqz0+dVNywzy7S89ZEBFwAXAUOBJcCYZN/M9loqcescRWtkEbEK+GQnxGJmedGWdgBvVcpdywMl/UbSSkkrJN0j6cDOCM7MMqh9HFkpWycppWn5C+B2YAiwH3AHcGs1gzKzbKvgmv0VUUoi6x4R/x0R25Lt50BTtQMzswzLy/ALSf2Sl/dLugy4jUJoHwN+1wmxmVlW5Wj4xSwKias94vO3OxfAV6sVlJllW/FV9DtXR3MtR3RmIGaWEyHoxOlHpShpZL+kQ4DRbNc3FhE/q1ZQZpZxeamRtZN0OTCeQiL7HfAB4HHAicxsb5WxRFbKXcvTKayfvSwiPkXhseZ9qhqVmWVbXu5abmdTRLRJ2iapN4VnzA2vclypOnL8Oi745hLq64L7b+3H7T8s+qDjvcKUG/bj97cOIgKO//hyTjl3CRvWNPDjC0eyalETA4Zt5sIfzadHX68pAFBXF1xz55M0r+jKFRe8K+1wKieDCyuWUiObKakvcD2FO5mzgT8V+5Ck4ZKmSnpO0lxJl+xZqJ2jri646NuL+fonR/BP40dywsQ17H/Q5rTDSt2i57vz+1sH8S+/eYpvTn6Spx7ux/JXm7jv2mEcfOxarnxsFgcfu5b7flTT/8aVZeJZi3n95e5ph1EVitK2zlI0kUXEhRGxJiJ+ApwMnJ00MYvZBnwxIkYD44CLJI3es3Crb+ThG1nyaheWLezKtpY6Hr2nL8ecsjbtsFK35MVuHHj4erp2a6O+AUaOW8us+/vz5IP9eO/pywF47+nLmT2lX5GS9g79B23hqONXM/mOwWmHUh0Za1ruMpFJOmLHDegHNCSvOxQRSyNidvJ6PYWlf4ZWKvBq6T+4hZVLury5v2ppIwOGtKQYUTYMG7mRF6b3YcMbDWzZVMfTU/eheWlX1q7qQt9Bhe+nz74trF3VpUhJe4fzv/YSN141graMdYpXStZqZB31kX2vg3MBnFjqRSQdABwOTNvJufOA8wCaqM1qeC3Y76BNnPqZRXz3k4fQtXsr+4/+C3V1b/0/VerMhVuya+z4ZtY0d2HB3F68e+yatMOpjoz1kXU0IPaESlxAUk/gLuDzEbFuJ9eZBEwC6K1+qf/71byskYH7bX1zf8CQFlYtbUwxouw4/ozlHH9GoRl555VvY58hW+gzYCtrljfSd1ALa5Y30nvA1iKl1L7RR6xj3InNHHX8ahq7tNG9Zytf+vf5XPXlUWmHVhmd3GwsRSmd/bstWVH2LuCWiLi7mteqlOfndGfoiK0MGr6FhsY2xk9cw5+neLQJwLpVhYTevLgrMx/oz7iJKxlz8moev7NwV/fxOwdx+Mmr0wwxE266egRnjT+aT500liu/OIqnp/WtnSTWLmN9ZFV70rgKi/zfAMyLiKurdZ1Ka2sV1/7zUL79i5epq4cpt/XjtRe82AfAD88fxYY3GqlvDM765kv06NPK31+4iGs/M4o//HIQ/Ydu4cIfz087TOsEytjCilVLZMCxwJnAM5LmJMe+FhGZXzljxiO9mfFI77TDyJyv3fXM3xzruc82vnLbsylEkw/PTO/LM9P7ph1G5WWsaVnKFCVRWOr6wIj4hqT9gcERMb2jz0XE47jv16zmdPYdyVKU0kf2I+AY4OPJ/nrg2qpFZGbZl7GlrktpWh4dEUdIehIgIt6Q5MFCZnuzjNXISklkLZLqSUKXNJDMPUPFzDpT1pqWpSSy/wR+Bewr6VsUVsP4elWjMrPsihzetYyIWyTNorCUj4DTIsJPGjfbm+WtRpbcpdwI/Gb7YxGxsJqBmVmG5S2RAffx14eQNAEjgOeBGlpgyczKkbU+slKW8Xl3RBya/DwIGEsJ65GZmZVCUr2kJyX9NtkfIWmapAWSflnKKImy51omS/McvRvxmlmtqOxcy0soLPPV7krg+xHxDuAN4JxiBZTSR/aF7XbrgCOAJSWHaGa1pYJ3LSUNAz4IfAv4QjKT6ETgE8lbbgauAH7cUTml9JH12u71Ngp9ZneVGa+Z1ZLSa1sDJM3cbn9SsnRXu/8Avsxf80x/YE1EbEv2F1HCgqwdJrJkIGyviPhSqVGbWW0TZXX2r4qII3dajvT3wIqImCVp/J7EtMtEJqkhIrZJOnZPLmBmNagydy2PBf5B0qkURkT0Bq4B+rbnH2AYsLhYQR119revbjFH0r2SzpT0kfZtD38BM8urEtfrL1Zri4ivRsSwiDgAOAN4JCI+CUylMIMI4GzgnmIhldJH1gQ0U+iAax9PFkAuVnw1syqo7hSlrwC3Sfo34EkKC7R2qKNEtm9yx/JZ/prA2mVsOJyZdaZKD4iNiEeBR5PXL1MYr1qyjhJZPdCTnS+O6ERmtjfLWAboKJEtjYhvdFokZpYPGXyKUkeJzMtUm9lOZW2uZUeJ7KROi8LM8iUviSwi/IBCM9up3C2saGb2FjnrIzMz+xsiex3oTmRmVj7XyMws7/J019LMbOecyMws1/L4ODgzs7/hGpmZ5Z37yMws/5zIdk1dGmkYPCztMDLrnENOTTuEzLtszmNph5Bp5//DhoqU4xqZmeVbUO2FFcvmRGZmZSnz4SOdwonMzMrnRGZmeafIViZzIjOz8nj1CzOrBe4jM7Pc8xQlM8s/18jMLNdKeIp4Z3MiM7PyOZGZWZ55QKyZ1QS1ZSuTOZGZWXk8jszMaoGHX5hZ/rlGZmZ5585+M8u3ADxp3Mzyzn1kZpZrHkdmZvkX4aalmeWfa2Rmln8ZS2R1aQdgZvmjKG3rsAxpuKSpkp6TNFfSJcnxfpIelPRi8nOfYvE4kZlZeQJojdK2jm0DvhgRo4FxwEWSRgOXAQ9HxEHAw8l+h5zIzKxslaiRRcTSiJidvF4PzAOGAhOBm5O33QycViwe95GZWfkqfNdS0gHA4cA0YFBELE1OLQMGFfu8E5mZla2Mu5YDJM3cbn9SREx6S1lST+Au4PMRsU7Sm+ciIqTiV3MiM7PylLeMz6qIOHJXJyU1Ukhit0TE3cnh5ZKGRMRSSUOAFcUu4j4yMyuLALVGSVuH5RSqXjcA8yLi6u1O3Qucnbw+G7inWEyukZlZ2Sr0pPFjgTOBZyTNSY59DfgOcLukc4DXgI8WK8iJzMzKU6EVYiPicQoVvJ05qZyynMh24rQzXub9E18nAl57qTff/+ahtGytTzuszKmrC66580maV3TligvelXY4qZt2w0Dm3N4PCQa+czMf+u5C7v/6cF6b1oOuvQrLRXzouwsZPHpTypHuqb1orqWkJuAxoGtynTsj4vJqXa9S+g/czIc+9iqfOeN4tm6p57Jvzeb4k5fw0H3D0w4tcyaetZjXX+5O956taYeSunXLGplx8wDOnzKfxqbg7ovfxtzfFAakn3TZEg4+dW3KEVZW1uZaVrOzfwtwYkQcBowBJkgaV8XrVUx9fdClayt19W10bWqleVVT2iFlTv9BWzjq+NVMvmNw2qFkRlur2La5jrZt0LKpjl6DWtIOqXraV8AotnWSqtXIIiKADcluY7JlLI//reaVTdx9y4HcdM8jbN1Sz+xpA3hy2sC0w8qc87/2EjdeNYJuPVwbA+g9uIVx567gB+8dTWNTMOK96zjw79Yz9959ePR7Q3j8B4M54D3rOeHLS2nomvk/g44FRe9IdraqDr+QVJ/cjVgBPBgR06p5vUro2auFccct59MfPoEzP3gSTd1aOWHCorTDypSx45tZ09yFBXN7pR1KZmxaW88LD/Xhot8/x+f+9Cwtm+p55tf7MP7/LOGCh+bzqV+/wKa1Dfzpun3TDrUyosStk1Q1kUVEa0SMAYYBYyUdsuN7JJ0naaakmVtb0+8EHXPUKpYv6ca6NV1pba3jj1MHc/C730g7rEwZfcQ6xp3YzH89PJ2vfG8+hx69hi/9+/y0w0rVq0/0pO+wrfTo30p9I4w8ZQ2LZvWg177bkKCha3DY6atZ8lT3tEOtCEWUtHWWTrlrGRFrJE0FJgDP7nBuEjAJoE/XQanXV1cub2LkIWvo2rWVLVvqOOyoVSyY1yftsDLlpqtHcNPVIwB499g1/M9PL+aqL49KOap09d6vhcVzutOySTQ0Ba/+sRdD3r2R9Ssa6LXvNiLg+Sl9GPjOzWmHWhl70V3LgUBLksS6AScDV1brepXy/Nx9eOKRIVzzsz/Q2ipefqEP9/96/7TDsowbOmYjoyas5YYPjaSuIRg0ehOHn9HMbZ8+kI3NhT+zQQdv4gP/trRISTkQwF708JEhwM2S6ik0YW+PiN9W8XoVc8v17+SW69+Zdhi58Mz0vjwzvW/aYWTC8Zcu4/hLl73l2D/e8lJK0VSP6NxmYymqedfyaQrLcphZrWnLVpXMI/vNrDx7WdPSzGrUXtO0NLMa5kRmZvm2F00aN7Ma1f4UpQxxIjOzsrmPzMzyz4nMzHItgDYnMjPLNXf2m1ktcCIzs1wLoDVbQ/udyMysTAHhRGZmeeempZnlmu9amllNcI3MzHLPiczMci0CWrP1GEAnMjMrn2tkZpZ7TmRmlm/hu5ZmlnMB4QGxZpZ7nqJkZrkW4cfBmVkNcGe/meVduEZmZvnmhRXNLO88adzM8i6AyNgUpbq0AzCznIlkYcVStiIkTZD0vKQFki7b3ZBcIzOzskUFmpaS6oFrgZOBRcAMSfdGxHPlluUamZmVrzI1srHAgoh4OSK2ArcBE3cnHEWG7j5IWgm8lnYc2xkArEo7iAzz91Nc1r6jt0XEwD0pQNIDFH6vUjQBm7fbnxQRk5JyTgcmRMS5yf6ZwNERcXG5MWWqabmnX3ClSZoZEUemHUdW+fsprha/o4iYkHYMO3LT0szSshgYvt3+sORY2ZzIzCwtM4CDJI2Q1AU4A7h3dwrKVNMygyalHUDG+fspzt/RLkTENkkXA5OBeuDGiJi7O2VlqrPfzGx3uGlpZrnnRGZmuedEthOSbpS0QtKzaceSRZKGS5oq6TlJcyVdknZMWSKpSdJ0SU8l38+/ph1TrXMf2U5IOg7YAPwsIg5JO56skTQEGBIRsyX1AmYBp+3O1JJaJElAj4jYIKkReBy4JCL+nHJoNcs1sp2IiMeA1WnHkVURsTQiZiev1wPzgKHpRpUdUbAh2W1MNtcYqsiJzPaIpAOAw4FpKYeSKZLqJc0BVgAPRoS/nypyIrPdJqkncBfw+YhYl3Y8WRIRrRExhsJo9bGS3EVRRU5ktluSvp+7gFsi4u6048mqiFgDTAUyNz+xljiRWdmSzuwbgHkRcXXa8WSNpIGS+iavu1FYb2t+qkHVOCeynZB0K/AnYKSkRZLOSTumjDkWOBM4UdKcZDs17aAyZAgwVdLTFOYTPhgRv005pprm4RdmlnuukZlZ7jmRmVnuOZGZWe45kZlZ7jmRmVnuOZHliKTWZKjDs5LukNR9D8q6KXmKDZJ+Kml0B+8dL+k9u3GNVyX9zdN2dnV8h/ds6Oj8Tt5/haQvlRuj1QYnsnzZFBFjkhU5tgIXbH9S0m4tXR4R5xZZuWI8UHYiM+ssTmT59QfgHUlt6Q+S7gWeSyYrf1fSDElPSzofCqPxJf0weTz9Q8C+7QVJelTSkcnrCZJmJ2tpPZxMCr8AuDSpDf5dMnL9ruQaMyQdm3y2v6QpyRpcPwVU7JeQ9GtJs5LPnLfDue8nxx+WNDA59nZJDySf+YOkURX5Ni3X/PCRHEpqXh8AHkgOHQEcEhGvJMlgbUQcJakr8ISkKRRWqBgJjAYGAc8BN+5Q7kDgeuC4pKx+EbFa0k+ADRFxVfK+XwDfj4jHJe1P4eERBwOXA49HxDckfRAoZUbEp5NrdANmSLorIpqBHsDMiLhU0r8kZV9M4WEeF0TEi5KOBn4EnLgbX6PVECeyfOmWLA0DhRrZDRSafNMj4pXk+PuBQ9v7v4A+wEHAccCtEdEKLJH0yE7KHwc81l5WROxqTbb3AaMLUy4B6J2shHEc8JHks/dJeqOE3+lzkj6cvB6exNoMtAG/TI7/HLg7ucZ7gDu2u3bXEq5hNc6JLF82JUvDvCn5g/7L9oeAz0bE5B3eV8m5kHXAuIjYvJNYSiZpPIWkeExEbJT0KNC0i7dHct01O34HZu4jqz2Tgc8ky+wg6Z2SegCPAR9L+tCGACfs5LN/Bo6TNCL5bL/k+Hqg13bvmwJ8tn1H0pjk5WPAJ5JjHwD2KRJrH+CNJImNolAjbFcHtNcqP0GhyboOeEXS/0quIUmHFbmG7QWcyGrPTyn0f81W4eEp11Goef8KeDE59zMKq3u8RUSsBM6j0Ix7ir827X4DfLi9sx/4HHBkcjPhOf569/RfKSTCuRSamAuLxPoA0CBpHvAdCom03V8oLEj4LIU+sG8kxz8JnJPENxeYWMJ3YjXOq1+YWe65RmZmuedEZma550RmZrnnRGZmuedEZma550RmZrnnRGZmuff/ASKwbsOmzYasAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################################\n",
    "# SINGLE TRIAL EEGNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Configure\n",
    "retrain_model = False\n",
    "tensorboard_name = \"EEGNet_singlesession_C0_100hz\"\n",
    "best_model_filename = f\"./saved_variables/6/EEGNet/{tensorboard_name}\"\n",
    "\n",
    "# Get train/test split of data from one session\n",
    "X_train, X_test, y_train, y_test = train_test_split(mne_fixed_window_epochs_data, ohe_labels, \n",
    "                                                    test_size = 0.3,\n",
    "                                                    shuffle= True,\n",
    "                                                    stratify= ohe_labels,                                                    \n",
    "                                                    random_state=98)\n",
    "\n",
    "\n",
    "# Train on train/test split of data from one session\n",
    "## Note: the model is forced to use GPU, if GPU is not available replace with what is available e.g. /cpu:0\n",
    "if (retrain_model): # Retrain or not\n",
    "    with tf.device('/gpu:0'):\n",
    "        keras_eegnet_model.fit(\n",
    "            x= X_train,\n",
    "            y= y_train,\n",
    "            batch_size= 128, # Default: 32\n",
    "            epochs= 500, # Default: 500 (EEGNet paper)\n",
    "            verbose= 1, # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "            #callbacks= [tensorboard_callback(\"EEGNet_raw_signal_0.5s_100samps_50kernlen_02nr\")], # To be used for TF Board\n",
    "            callbacks= [tensorboard_callback(tensorboard_name),\n",
    "                        lowest_loss_model_save_callback(best_model_filename),\n",
    "                        highest_accuracy_model_save_callback(best_model_filename)],\n",
    "            validation_split= 0.3,\n",
    "            shuffle= True,\n",
    "            sample_weight= None, # Can be interesting due to time series\n",
    "            use_multiprocessing=True, # Done for faster speed\n",
    "            workers= 4 # Done for faster speed\n",
    "            )\n",
    "\n",
    "# Convert labels back to original\n",
    "y_test = ohe.inverse_transform(y_test)\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"Results for lowest loss model\")\n",
    "keras_eegnet_model = load_lowest_loss_model(best_model_filename)\n",
    "\n",
    "y_pred = keras_eegnet_model.predict(X_test)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained EEGNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"\\n\\nResults for highest accuracy model\")\n",
    "keras_eegnet_model = load_highest_accuracy_model(best_model_filename)\n",
    "\n",
    "y_pred = keras_eegnet_model.predict(X_test)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained EEGNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Remove unused variables\n",
    "del X_train\n",
    "del X_test\n",
    "del y_train\n",
    "del y_test\n",
    "del accuracy\n",
    "del y_pred\n",
    "del retrain_model\n",
    "del tensorboard_name\n",
    "del best_model_filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "18770e4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAUP\n",
    "####################################################\n",
    "\n",
    "# delete unused variables\n",
    "del keras_eegnet_model\n",
    "del mne_fixed_window_epochs_data\n",
    "del ohe\n",
    "del ohe_labels\n",
    "del mne_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8281928",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Fixed window classification: Multi trial training with unseen trial testing | EEGNet | three class MI task | 100Hz input signal\n",
    "\n",
    "Having the best performance on single trials, it is expected EEGNet will perform very well on the multi trial training and single trial testing experiments as well.\n",
    "We test this here.\n",
    "The scores given are best loss/ best accuracy.\n",
    "\n",
    "**Remember the results for subject `C`**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|\n",
    "| 0              | 1 + 2           | 0.782                        | 0.916 / 0.920             |\n",
    "| 1              | 0 + 2           | 0.635                        | 0.852 / 0.860             |\n",
    "| 2              | 1 + 2           | 0.347                        | 0.607 / 0.594             |\n",
    "| 0              | 1               | 0.667                        | 0.891 / 0.860             |\n",
    "| 1              | 0               | 0.662                        | 0.794 / 0.783             |\n",
    "\n",
    "For test index 2 trained on index 0 and 1, EEGNet is the first to somewhat accurately predict the third neutral class as well.\n",
    "\n",
    "**For subject `B` these results are**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|\n",
    "| 0              | 1 + 2           | 0.409                        | 0.498 / 0.489             |\n",
    "| 1              | 0 + 2           | 0.497                        | 0.640 / 0.654             |\n",
    "| 2              | 1 + 2           | 0.502                        | 0.677 / 0.660             |\n",
    "\n",
    "**For subject `E` the results are**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|\n",
    "| 0              | 1 + 2           | 0.398                        | 0.714 / 0.713             |\n",
    "| 1              | 0 + 2           | 0.371                        | 0.667 / 0.646             |\n",
    "| 2              | 1 + 2           | 0.475                        | 0.734 / 0.688             |\n",
    "\n",
    "Again, we see that the DL approach using EEGNet far outperforms the CSP approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "757ebb42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 0 ... 666799  =      0.000 ...  3333.995 secs...\n",
      "Reading 0 ... 681199  =      0.000 ...  3405.995 secs...\n",
      "Reading 0 ... 669399  =      0.000 ...  3346.995 secs...\n",
      "Loaded test epochs and extracted labels\n",
      "Loaded train epochs and extracted labels\n",
      "Loaded train epochs and extracted labels\n",
      "\n",
      "\n",
      "Concatenated the train epochs\n",
      "Total amount of train labels: 1919\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Lennert\\AppData\\Local\\Temp\\ipykernel_16324\\2395505687.py:29: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.\n",
      "  mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
      "C:\\Users\\Lennert\\AppData\\Local\\Temp\\ipykernel_16324\\2395505687.py:34: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.\n",
      "  mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Got the test epochs\n",
      "Total amount of train labels: 960\n",
      "OHE test and train labels, train labels:\n",
      "[[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]]\n",
      "Train labels OHE match regular labels: True\n",
      "Test labels OHE match regular labels: True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# PREPPING THE DATA\n",
    "####################################################\n",
    "\n",
    "# Get data and choose test trial\n",
    "mne_raws = load_mne_raws(subject= \"C\")\n",
    "test_trial = mne_raws[0]\n",
    "\n",
    "# Init variables\n",
    "epochs = []\n",
    "labels_train = []\n",
    "\n",
    "for single_mne_raw in mne_raws:\n",
    "    # Load epochs from raw\n",
    "    mne_fixed_window_epochs = CLA_dataset.get_usefull_epochs_from_raw(single_mne_raw, start_offset=-1.5, end_offset=1.5)['task/neutral', 'task/left', 'task/right']\n",
    "    with io.capture_output():\n",
    "        mne_fixed_window_epochs.load_data()\n",
    "    \n",
    "    if (single_mne_raw == test_trial):\n",
    "        mne_fixed_window_epochs_test = mne_fixed_window_epochs\n",
    "        labels_test = mne_fixed_window_epochs.events[:, -1]\n",
    "        print(\"Loaded test epochs and extracted labels\")\n",
    "    else:  \n",
    "        epochs.append(mne_fixed_window_epochs)\n",
    "        labels_train.extend(mne_fixed_window_epochs.events[:, -1])\n",
    "        print(\"Loaded train epochs and extracted labels\")\n",
    "    \n",
    "# Make single epoch object for training\n",
    "mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
    "print(\"\\n\\nConcatenated the train epochs\")\n",
    "print(f\"Total amount of train labels: {len(labels_train)}\")\n",
    "    \n",
    "# Show test epoch object\n",
    "mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
    "print(\"\\n\\nGot the test epochs\")\n",
    "print(f\"Total amount of train labels: {len(labels_test)}\")\n",
    "\n",
    "# Go to 2D representation\n",
    "labels_train = np.array(labels_train).reshape(-1, 1)\n",
    "labels_test = np.array(labels_test).reshape(-1, 1)\n",
    "\n",
    "# One Hot Encode the labels\n",
    "ohe = OneHotEncoder()\n",
    "ohe_labels_train = ohe.fit_transform(labels_train).toarray()\n",
    "ohe_labels_test = ohe.transform(labels_test).toarray()\n",
    "\n",
    "\n",
    "# Show ohe labels\n",
    "print(f\"OHE test and train labels, train labels:\\n{ohe_labels_train[:10]}\")\n",
    "\n",
    "# Validate OHE\n",
    "print(f\"Train labels OHE match regular labels: {np.array_equal(ohe.inverse_transform(ohe_labels_train), labels_train)}\")\n",
    "print(f\"Test labels OHE match regular labels: {np.array_equal(ohe.inverse_transform(ohe_labels_test), labels_test)}\")\n",
    "\n",
    "# Get training data\n",
    "mne_fixed_window_epochs_train_data = mne_fixed_window_epochs_train.get_data(tmin=0.2, tmax=0.7)\n",
    "mne_fixed_window_epochs_test_data = mne_fixed_window_epochs_test.get_data(tmin=0.2, tmax=0.7)\n",
    "\n",
    "# Fix scaling sensitivity as MNE stores as data * 10e-6\n",
    "mne_fixed_window_epochs_train_data = mne_fixed_window_epochs_train_data * 1000000\n",
    "mne_fixed_window_epochs_test_data = mne_fixed_window_epochs_test_data * 1000000\n",
    "\n",
    "# Remove unused variables\n",
    "del epochs\n",
    "del mne_fixed_window_epochs\n",
    "del test_trial\n",
    "del single_mne_raw\n",
    "del mne_raws\n",
    "del mne_fixed_window_epochs_train\n",
    "del mne_fixed_window_epochs_test\n",
    "del labels_train\n",
    "del labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f4cef7b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_2 (InputLayer)        [(None, 21, 100, 1)]      0         \n",
      "                                                                 \n",
      " conv2d_1 (Conv2D)           (None, 21, 100, 8)        400       \n",
      "                                                                 \n",
      " batch_normalization_3 (Batc  (None, 21, 100, 8)       32        \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " depthwise_conv2d_1 (Depthwi  (None, 1, 100, 16)       336       \n",
      " seConv2D)                                                       \n",
      "                                                                 \n",
      " batch_normalization_4 (Batc  (None, 1, 100, 16)       64        \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_2 (Activation)   (None, 1, 100, 16)        0         \n",
      "                                                                 \n",
      " average_pooling2d_2 (Averag  (None, 1, 25, 16)        0         \n",
      " ePooling2D)                                                     \n",
      "                                                                 \n",
      " spatial_dropout2d_2 (Spatia  (None, 1, 25, 16)        0         \n",
      " lDropout2D)                                                     \n",
      "                                                                 \n",
      " separable_conv2d_1 (Separab  (None, 1, 25, 16)        512       \n",
      " leConv2D)                                                       \n",
      "                                                                 \n",
      " batch_normalization_5 (Batc  (None, 1, 25, 16)        64        \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_3 (Activation)   (None, 1, 25, 16)         0         \n",
      "                                                                 \n",
      " average_pooling2d_3 (Averag  (None, 1, 3, 16)         0         \n",
      " ePooling2D)                                                     \n",
      "                                                                 \n",
      " spatial_dropout2d_3 (Spatia  (None, 1, 3, 16)         0         \n",
      " lDropout2D)                                                     \n",
      "                                                                 \n",
      " flatten (Flatten)           (None, 48)                0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 3)                 147       \n",
      "                                                                 \n",
      " softmax (Activation)        (None, 3)                 0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,555\n",
      "Trainable params: 1,475\n",
      "Non-trainable params: 80\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE EEGNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Create the TensorFlow Keras model\n",
    "keras_eegnet_model = EEGNet(\n",
    "    nb_classes = 3, # int, number of classes to classify. \n",
    "    Chans = 21, # number of channels in the EEG data. \n",
    "    Samples = 100, # number of time points in the EEG data. (default: 128)\n",
    "    dropoutRate = 0.5, # dropout fraction. (default: 0.5)\n",
    "    kernLength = 50, # length of temporal convolution in first layer. Suggested: half the sampling rate. (default: 64)\n",
    "    F1 = 8, # number of temporal filters. (default: 8)\n",
    "    F2 = 16, # number of pointwise filters. (default: 16)\n",
    "    D = 2, # number of spatial filters to learn within each temporal convolution. (default: 2)\n",
    "    norm_rate = 0.25, # Normalisation rate. (default: 0.25)\n",
    "    dropoutType = 'SpatialDropout2D' # Either SpatialDropout2D or Dropout, passed as a string. (default: Dropout)\n",
    "    )\n",
    "\n",
    "# Compile the model so it can be fitted\n",
    "# Loss and optimizer from EEGNet paper\n",
    "keras_eegnet_model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics=[\"accuracy\"])\n",
    "\n",
    "# Show summary of the model\n",
    "keras_eegnet_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0fecd45c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for lowest loss model\n",
      "Trained EEGNet on single session using train/test split and got accuracy of: 0.915625\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhCUlEQVR4nO3deXwV5fn38c+VEAiETQhg2EEQRSxg0aJURKmKVkX7tFZtFa0tbbVVW7Uu7a/a9mef2roUtdri8oj7hlbcwKUqWldAUARREJAdwhIIhCUn1/PHTDAqnJxDcjLnTL5vX/Ny5p45M1fOK1y5l5l7zN0REYmjvKgDEBHJFCU4EYktJTgRiS0lOBGJLSU4EYmtJlEHUFNxu3zv2a0g6jCy1sfvt4g6hKxnefqbnUxFVTnbfavV5RzHHlnka9clUjp2+vvbprj7qLpcry6yKsH17FbAO1O6RR1G1jq286CoQ8h6eS2Kog4hq7215ek6n6N0XYK3p3RN6diCkgXFdb5gHejPnYikyUl4VUpLMmZWaGbvmNksM/vQzP4Qlvcys7fNbL6ZPWxmTcPyZuH2/HB/z9oiVYITkbQ4UIWntNRiG3CUuw8EBgGjzGwocC1wo7v3AdYD54bHnwusD8tvDI9LSglORNJWleJ/yXigPNwsCBcHjgIeC8snACeH66PDbcL9I80saX9iVvXBiUj2c5wdtTQ/ayg2s2k1tse7+/jqDTPLB6YDfYB/AAuADe5eGR6yFOgSrncBlgC4e6WZlQHtgdLdXVwJTkTS4kCi9uZntVJ3H7Lbc7kngEFm1hZ4AtivzgHWoCaqiKStnvrgdnL3DcDLwKFAWzOrrnx1BZaF68uAbgDh/jbA2mTnVYITkbQ4kHBPaUnGzDqENTfMrDlwNDCXINF9NzxsDPBkuD4p3Cbc/x+vZTokNVFFJG0p98AlVwJMCPvh8oBH3P1pM5sDPGRm/wu8B9wZHn8ncK+ZzQfWAafVdgElOBFJi+Pp9MHt/jzu7wODd1H+KXDILsq3At9L5xpKcCKSFnfYkSPz5CrBiUiajAR1epy1wSjBiUhaHKhSDU5E4ko1OBGJpeBGXyU4EYkhB3Z4btxCqwQnImlxjESOPCOgBCciaatyNVFFJIbUByciMWYk1AcnInEUzOirBCciMeRubPf8qMNIiRKciKStSn1wIhJHwSCDmqgiEksaZBCRmNIgg4jEWkI3+opIHDnGDs+N1JEbUYpI1tAgg4jElmNqoopIfGmQIctt32pc/J0+7NieR6ISDv92GWddupKVnzXlzz/vwcb1Teh74BZ+c/NnFDQN5md+dVJb7rt+bzCnd/+tXHHr4oh/imgUNKvi+sfnU9DUyW/ivPZMW+69bu+ow4rcr/7vfA45ch0b1hbw828HL4s686LPOHTkOqocytYWcP1lfVm3umnEkdaNO7pNxMzuAk4AVrv7gExdZ08VNHP++ugCmhdVUbkDfn1yXw4+aiMTx3fgOz9Zw4iTNzDusq5MfrAdJ45Zy7JPm/LwzR254clPaNU2wYbSRvu3gR3bjN98bx+2bsknv4lzw7/n8+5/WvHRjKKoQ4vUC493YNK9e3PJ3z7ZWTbxjs7c+/fuAJx01grO+MUSbvn9PlGFWC+CQYbceFQrk2n4bmBUBs9fJ2bQvCh4fW3lDiOxwzCDWa+34vATNgBw9PfW8ebkNgA8d397Tjy7lFZtEwC0La6MJO7sYGzdEvyCNylw8gucWl5i3ijMfrcNm8q++IdvS/nn24XNE9TD60SzQoK8lJaoZawa4u5Tzaxnps5fHxIJ+MWx/Vi+qCknnl1KSY9tFLVJkB9+K8UlOyhdWQDA0k8LAfjVSX2oqjJ+ePFKDj5yU1ShRy4vz7llysd07rmdp+5uz7z3GnftLZkxv1rMyFPWsHlTPpefmXWNmbQ5ljMTXkafYiOUnw+3vTiP+6fPYd7MFiyZX7jbYxMJWLawGX+bOJ8rbl3M3y/pRnlZblTTM6Gqyjjv6H784Ov96TdoCz36VUQdUtaacGMPzho+hJcndeDEH66IOpx6kSs1uMgjMLOxZjbNzKatWZuIJIaWbRIMPKycudNbsLksn0TY+ixdUUDx3juAoDY39JiNNCmAvbtvp+s+21i2MLc7i+vD5o35zHqjZaOuzabq5UkdGHbs2qjDqLPgvah5KS3JmFk3M3vZzOaY2YdmdmFYfrWZLTOzmeFyfI3PXGFm881snpkdW1uskSc4dx/v7kPcfUiH9g1XI9qwNn9nDWxbhTFjaiu69d3GwGHlvPZ0WwBeeLQdhx5bBsBho8p4/82WAJStzWfpgmaUdN/eYPFmkzbtKilqHfwxalpYxUHDy5PWfhuzzj0+r9ke+q11LP20eYTR1JfgzfapLLWoBC529/7AUOB8M+sf7rvR3QeFy7MA4b7TgAMI+vdvNbOkSaPRDgWuW1XAdRd2p6rKqKqC4SduYOjRG+mx71b+/PMe3P3XEvoMqODY09cBMGTEJma82oqfHLEfefnOT/5nOa3bRVPjjFq7Tju4ZNxn5OVBXh5MfaoNb7/YOuqwInfZjR/ztUPKaL1XJfe+No17x3Xj4BHr6dqrAq8yVi9vxs2/7x11mHUWvDaw7pURd18BrAjXN5nZXKBLko+MBh5y923AQjObDxwCvLm7D5hnaPjLzB4ERgDFwCrgKne/M9lnhgws9HemdMtIPHFwbOdBUYeQ9fKKNNiRzFtbnqYsUVqnEYIuB7T18x75ZkrH/m7AM9PdfUhtx4UDklOBAcCvgbOBjcA0glreejO7BXjL3e8LP3Mn8Jy7P7a782asierup7t7ibsXuHvX2pKbiOSOhOeltADF1X3s4TL2y+cys5bAROAid98I3AbsAwwiqOFdv6dxNtomqojsmWA+uJQrgaXJanBmVkCQ3O5398cB3H1Vjf23A0+Hm8uAmk28rmHZbkU+yCAiucbSqcHt/ixmBtwJzHX3G2qUl9Q47BRgdrg+CTjNzJqZWS+gL/BOsmuoBiciaQluE6mXG32HAWcCH5jZzLDsSuB0MxsUXmoR8FMAd//QzB4B5hCMwJ7v7klH+pTgRCQt9fUsqru/Drts6z6b5DPXANekeg0lOBFJm6ZLEpFYCqZLyo1nUZXgRCRtufKwvRKciKQlmE1ETVQRiaHgUS0lOBGJJdXgRCTG0niSIVJKcCKSFo2iikisqYkqIrGUS+9kUIITkbQ4UKkanIjElZqoIhJPriaqiMRUmhNeRkoJTkTSphqciMRSPU54mXFKcCKSFseorNIgg4jElPrgRCSeXE1UEYkp9cGJSKwpwYlILDlGQoMMIhJXGmQQkVhyDTKISJy5EpyIxJMetheRGFMNbg98/EELRnUfEnUYWWvi0tejDiHrfbfX8KhDyGruXg/ngERV3ROcmXUD7gE6EdxeN97dx5lZO+BhoCewCDjV3debmQHjgOOBLcDZ7j4j2TVyY6xXRLJKFZbSUotK4GJ37w8MBc43s/7A5cBL7t4XeCncBjgO6BsuY4HbaruAEpyIpMUJmqipLEnP476iugbm7puAuUAXYDQwITxsAnByuD4auMcDbwFtzawk2TWyqokqIrkgrUGGYjObVmN7vLuP/8oZzXoCg4G3gU7uviLctZKgCQtB8ltS42NLw7IV7IYSnIikLY2uvFJ3T9qxbmYtgYnARe6+Mehqq76Ou5ntccehmqgikrb6aKICmFkBQXK7390fD4tXVTc9w/+vDsuXAd1qfLxrWLZbSnAikpZgFDUvpSWZcFT0TmCuu99QY9ckYEy4PgZ4skb5WRYYCpTVaMrukpqoIpK2erjbBGAYcCbwgZnNDMuuBP4CPGJm5wKLgVPDfc8S3CIyn+A2kXNqu4ASnIikrT5u9HX312G395KM3MXxDpyfzjWU4EQkLU5q/WvZQAlORNJWPy3UzFOCE5H0OHg9PKrVEJTgRCRtaqKKSGzV0yhqxu02wZnZzSRparv7BRmJSESyWvWzqLkgWQ1uWpJ9ItJYOZDrCc7dJ9TcNrMW7r4l8yGJSLbLlSZqrY9qmdmhZjYH+CjcHmhmt2Y8MhHJUoZXpbZELZVnUf8OHAusBXD3WYCmTRVpzDzFJWIpjaK6+5KaU5gAicyEIyJZz+MxyFBtiZkdBng4tcmFBDNvikhjlQW1s1Sk0kT9GcEDrl2A5cAg0nzgVUTixlJcolVrDc7dS4EfNEAsIpIrqqIOIDWpjKL2NrOnzGyNma02syfNrHdDBCciWaj6PrhUloil0kR9AHgEKAE6A48CD2YyKBHJbu6pLVFLJcG1cPd73b0yXO4DCjMdmIhksVy/TSR8uzTAc2Z2OfAQQcjfJ5g6WEQaqyxofqYi2SDDdIKEVv2T/LTGPgeuyFRQIpLd9vxFfg0r2bOovRoyEBHJEW6QBY9hpSKlJxnMbADQnxp9b+5+T6aCEpEsl+s1uGpmdhUwgiDBPQscB7wOKMGJNFY5kuBSGUX9LsErvFa6+znAQKBNRqMSkeyW66OoNVS4e5WZVZpZa2A10C3DcUWma++tXPGPT3du7919G/fe0Jl/39kpwqgaXunyptx04T6UlRaAwdFnrOaEH69k0ZwW/OvyXmzdnE+Hbtu46Ob5tGiVYMd241+X92LBrJZYnvOjPyxmwGEbo/4xInPKuSsZddoa3I1FHzXn+kt7sWNbKvWJHBCHCS9rmGZmbYHbCUZWy4E3a/uQmXUjaMZ2IvhKxrv7uD0PtWEs/bSQ84/rD0BennPfO+/zxuS20QYVgfx85+zfL6b3gVuoKM/j0uMOZODwMm69tDdjfreYAw7dxEsPdeDJf5Zw+qVLefGBjgDc+NL7lJU24X/P3I9rn5lNXkz+TaejfaftjD5nFWNHHsj2bXlc+Y/5jDhxHS88Vhx1aPUmV0ZRa/31c/fz3H2Du/8TOBoYEzZVa1MJXOzu/YGhwPlm1r9u4TasQcM2seKzZqxe1izqUBrcXp120PvAYALn5i2r6Nq3gnUrm7Li00L6D90EwMDhZbz1bHC75NJPmu+ssbUprqSodYIFs4qiCT4L5Oc7TQuryMt3mjWvYu2qgqhDql850kTdbYIzs4O+vADtgCbhelLuvsLdZ4TrmwimWOpSX4E3hCNOWscrT7ar/cCYW72kGQtnF9F3cDnd9q3gnSl7AfDG0+0oXR4k/x77b2HaC3uRqIRVnzVjwQdFO/c1NmtXNeWx8Xtz75uzeODdmWzelM+M1+LVbW2e2hK1ZE3U65Psc+CoVC9iZj2BwcDbu9g3FhgLUEiLVE+ZcU0Kqhh69Ab+37U5lZPrXcXmPP42ti/nXL2IFq0SnHf9Au76fU8eG9eFg49eT5OCYFqJkaetZtn85vzm+APp0HUb/b6+ibz8LPgNj0DL1pUceswGzv7m1yjfmM9vb13AUaeU8p8n4tNEra8+ODO7CzgBWO3uA8Kyq4GfAGvCw65092fDfVcA5xJMunuBu09Jdv5kN/oeWefog4BaAhOBi9z9K73O7j4eGA/QOq9d1vyLGDJiI/Nnt2BDacyaFmmo3GH8bey+HH5KKUOPXw9A1z5b+f0DHwGw/NNCpr8U1Obym8A5Vy/e+dkrRx9A595bGz7oLDD4mxtZtaQZZeuC353/Tt6L/b9eHp8EV7/Nz7uBW/jqbWc3uvt1NQvCLq7TgAMIJv540cz2dffdzjCe0S7gcAbgicD97v54Jq9V30aMbtzNU3e49ZLedO1TwUljV+4sLysN/iZWVcFj47pwzJmrANhWkcfWLcGv06ypbchr4nTbt6LhA88Cq5c3Zb/B5TQrTADOoGEbWTK/edRh1a966oNz96nAuhSvOhp4yN23uftCYD5wSLIPZOzN9ha8xOFOYK6735Cp62RCs+YJDjp8Izdd0SPqUCLz0buteHViB7rvt5mLjzkQgDMuW8KKhYVMnhDcMvON49Zx1PeDVkRZaQF/+sF+WB6023s7F4ybH1nsUZs3syWvPduOW56ZQyJhLPiwBc890CHqsOqVpT7hZbGZ1XzH8viw1VabX5jZWQTvZ77Y3dcT9OG/VeOYpdTSr5+xBAcMA84EPjCzmWHZzrZ0NttWkc+pAwdFHUak9j9kExOXvrXLfSf8eOVXyjp228bNU2dlOqyccd+NXbjvxhj336beRC119yFpnv024E/hVf5EMB7wozTPAaT2qJYRTFne293/aGbdgb3d/Z1kn3P318mGSdlFpF5leoTU3VftvJbZ7cDT4eYyvviQQdewbLdS6YO7FTgUOD3c3gT8I9VgRSSGMjhluZmV1Ng8BZgdrk8CTjOzZmbWC+gLJK1opdJE/Ya7H2Rm7wG4+3oza7oHcYtIXNRTDc7MHiSYzKPYzJYCVwEjzGxQeJVFhHNRuvuHZvYIMIfgQYLzk42gQmoJboeZ5YcXw8w6kDPv1BGRTKivJqq7n76L4juTHH8NcE2q508lwd0EPAF0NLNrCGYX+V2qFxCRmPG0RlEjlcp7Ue83s+kEUyYZcLK76832Io1Z1tySn1wqo6jdgS3AUzXL3P2zTAYmIlksLgkOeIbPXz5TCPQC5hE8LiEijVA2PEifilSaqAfW3A5nEjkvYxGJiNSTtJ9kcPcZZvaNTAQjIjkiLjU4M/t1jc084CBgecYiEpHsFqdRVKBVjfVKgj65iZkJR0RyQhxqcOENvq3c/ZIGikdEspwRg0EGM2vi7pVmNqwhAxKRHJDrCY7gIdaDgJlmNgl4FNhcvTPXJrAUkXqSJe9bSEUqfXCFwFqCdzBU3w/ngBKcSGMVg0GGjuEI6mw+T2zVciR/i0gmxKEGlw+0ZNeTVubIjyciGZEjGSBZglvh7n9ssEhEJDdkyUudU5EswWm6cRHZpTg0UUc2WBQikltyPcG5e6rvKhSRRiZOj2qJiHwuJn1wIiJfYeROB70SnIikTzU4EYmrOIyiiojsmhKciMRSzCa8FBH5ItXgRCSu1AcnIvGlBJc+y29CXvt2UYeRtb4/9P9EHULW+/Xc56MOIatdMHpLvZynvmpwZnYXcAKw2t0HhGXtgIeBnsAi4FR3X29mBowDjid4Gf3Z7j4j2fnz6idMEWk0nGDCy1SW2t0NjPpS2eXAS+7eF3gp3AY4DugbLmOB22o7uRKciKSl+qUzqSy1cfepwJefex8NTAjXJwAn1yi/xwNvAW3NrCTZ+ZXgRCR9nuICxWY2rcYyNoWzd3L3FeH6SqBTuN4FWFLjuKVh2W5lVR+ciOQG85Q74UrdfcieXsfd3WzPe/xUgxOR9KRae9vzgYhV1U3P8P+rw/JlQLcax3UNy3ZLCU5E0lZffXC7MQkYE66PAZ6sUX6WBYYCZTWasrukJqqIpK2+HtUysweBEQR9dUuBq4C/AI+Y2bnAYuDU8PBnCW4RmU9wm8g5tZ1fCU5E0ldP98G5++m72fWVVya4uwPnp3N+JTgRSU/M3mwvIvJFSnAiEkfVN/rmAiU4EUmbVeVGhlOCE5H06K1aIhJnmtFXROJLNTgRiSsNMohIPDmQ+sP2kVKCE5G0qQ9ORGJJ98GJSHy5q4kqIvGlGpyIxJcSnIjElWpwIhJPDiRyI8MpwYlI2lSDE5H40iiqiMSVanAiEk+aLklE4soA0yCDiMRVGm+2j5QSnIikR03U3FLcaSsX/2k2e7XfjjtMntiVJx/szpnnzWfoEWuocihb15QbrjqAdWsKow43EsUdK7j46lm0bbcdByY/0Y1JD/fismveo2uPcgCKWlayubwJv/zh4dEG20A2Lm/C5Eu7srk0HzP42mnrOejsdTx1QVfWL2wKwLaN+TRrneCspz6t8bkC7h61D4desIaDf7w2qvDrQM+iYmaFwFSgWXidx9z9qkxdry4SCeOOG/ZlwUetad6ikpseeJsZb7fjsQk9uffWPgCcdPpnnDH2U265pn/E0UYjkTDuGLc/C+a1oXmLSsbd8zrvvVPMtb8dvPOYcy+cy5byxvM3M68JHHHFSjoN2Mr28jzuO7k3PYZt5sSblu485pU/d6JZqy/OLfTKNZ3oNby8ocOtV7kyipqXwXNvA45y94HAIGCUmQ3N4PX22PrSZiz4qDUAFVua8NnCIoo7bKNi8+f/WAubJ3C3qEKM3Pq1hSyY1wYIvqMlC1vSvsPWGkc4h39rBa8+3zmaACPQsmMlnQYE30HTllW022cbm1Z9/jvjDvOebcN+J5btLPvkhVa06baD9n23NXi89ap6RpHalohl7M+tuztQ/WeqIFyi/4lr0bGkgn36beKj2cE/5rPOn8/IE5azubwJl48dEnF02aFjyRZ699vIvA/b7iw7YPB6NqxryvIlRdEFFqGypQWsnlNIycCKnWXL3m1BUXEle/XcDsD2zXm8+69ivjthMdPuaB9VqHXnuTOKmskaHGaWb2YzgdXAC+7+diavV1eFzSv57XWzGH/dvjtrb/f8ow9jjhvOK8+VcOL3l0QcYfQKm1fy27/M4PYb+lOxuWBn+RHHLOfVKY2n9lbT9s15TDq/G0f+buUXmqMfPd2G/U74vPb2xk0d+Po5a2lalCPT4SbjKS61MLNFZvaBmc00s2lhWTsze8HMPgn/v9eehpnRBOfuCXcfBHQFDjGzAV8+xszGmtk0M5u2variK+doKPlNqvjtde/zynMlvPGfTl/Z//KzezNs5KoIIsse+flVXHntDF6e0pk3Xtl7Z3lefhWHjVjJ1BdLIowuGokdMOn8bux/Uhl9j920s7yqEj6Z0pp+3/48wa2c1Zypf+3E7Uf0Zcbd7XnntmLeu6ddFGHXmbmntKToSHcf5O7VTaTLgZfcvS/wUri9RxqkR9jdN5jZy8AoYPaX9o0HxgO0KegYUb3XueiqOSxZWMQT9/XYWdq5+2aWfxY0uYaOWMPSRY2z+RVwLvyfD1iysCX/fqD3F/YMPngtSxe3ZO3q5hHFFg13eP6KLrTvs40h535xNHTxf1vSrvc2WpVU7iw77aFFO9ffGNeBgqIqBp+1rqHCrV+Z7V8bDYwI1ycArwCX7cmJMjmK2gHYESa35sDRwLWZul5d9B+0gZEnrGDhxy25+aE3AZhwSx+OPXk5XXpsxquM1SsKueWa/SOONDr9B65n5PHLWPhJK26+7zUAJtzaj2lvdGT4Mct59fnGV3tbNr0Fc/7dluJ+W7nnxCDpf/Pi1fQeUc68Z1p/YXAhVhxIvZVdXN30DI0PKzU1z/a8mTnwr3BfJ3dfEe5fCXy1SZUi8wxlYjP7GkH2zSdoCj/i7n9M9pk2BR390OLvZSSeOLCCgtoPauQufOX5qEPIaheMXsjHH1TU6XaANkWdfWj/n6Z07PPTrp5eo+n5FWbWxd2XmVlH4AXgl8Akd29b45j17r5H/XCZHEV9Hxhc64Eiknuq6megxN2Xhf9fbWZPAIcAq8ysxN1XmFkJwSDlHsnoIIOIxFB1EzWVJQkzKzKzVtXrwDEEffSTgDHhYWOAJ/c01MZz27mI1Jt6eti+E/CEmUGQix5w98lm9i7wiJmdCywGTt3TCyjBiUj66iHBufunwMBdlK8FRtb5AijBiUjasuMxrFQowYlIevRWLRGJM014KSLxpQQnIrHkQJUSnIjEkgYZRCTOlOBEJJYcSOTGnHZKcCKSJgdXghORuFITVURiSaOoIhJrqsGJSGwpwYlILLlDIhF1FClRghOR9KkGJyKxpQQnIvHkGkUVkZhycN3oKyKxpUe1RCSW3OvttYGZpgQnIunTIIOIxJWrBici8aQJL0UkrvSwvYjElQOuR7VEJJZcE16KSIy5mqgiEls5UoMzz6LREDNbAyyOOo4aioHSqIPIYvp+apdt31EPd+9QlxOY2WSCnysVpe4+qi7Xq4usSnDZxsymufuQqOPIVvp+aqfvKFp5UQcgIpIpSnAiEltKcMmNjzqALKfvp3b6jiKkPjgRiS3V4EQktpTgRCS2lOB2wczuMrPVZjY76liykZl1M7OXzWyOmX1oZhdGHVM2MbNCM3vHzGaF388foo6psVIf3C6Y2XCgHLjH3QdEHU+2MbMSoMTdZ5hZK2A6cLK7z4k4tKxgZgYUuXu5mRUArwMXuvtbEYfW6KgGtwvuPhVYF3Uc2crdV7j7jHB9EzAX6BJtVNnDA+XhZkG4qCYRASU4qRMz6wkMBt6OOJSsYmb5ZjYTWA284O76fiKgBCd7zMxaAhOBi9x9Y9TxZBN3T7j7IKArcIiZqasjAkpwskfCvqWJwP3u/njU8WQrd98AvAxE9sB5Y6YEJ2kLO9HvBOa6+w1Rx5NtzKyDmbUN15sDRwMfRRpUI6UEtwtm9iDwJtDPzJaa2blRx5RlhgFnAkeZ2cxwOT7qoLJICfCymb0PvEvQB/d0xDE1SrpNRERiSzU4EYktJTgRiS0lOBGJLSU4EYktJTgRiS0luBxiZonwlozZZvaombWow7nuNrPvhut3mFn/JMeOMLPD9uAai8zsK29f2l35l44pT7Z/F8dfbWaXpBujxJsSXG6pcPdB4Qwn24Gf1dxpZnv0nlt3/3EtM4GMANJOcCJRU4LLXa8BfcLa1WtmNgmYEz7k/Tcze9fM3jezn0Lw9IGZ3WJm88zsRaBj9YnM7BUzGxKujzKzGeFcZi+FD9P/DPhVWHs8PLxTf2J4jXfNbFj42fZm9nw4B9odgNX2Q5jZv81seviZsV/ad2NY/pKZdQjL9jGzyeFnXjOz/erl25RY0pvtc1BYUzsOmBwWHQQMcPeFYZIoc/eDzawZ8F8ze55gxo9+QH+gEzAHuOtL5+0A3A4MD8/Vzt3Xmdk/gXJ3vy487gHgRnd/3cy6A1OA/YGrgNfd/Y9m9m0glSdAfhReoznwrplNdPe1QBEwzd1/ZWa/D8/9C4KXuPzM3T8xs28AtwJH7cHXKI2AElxuaR5OwQNBDe5OgqbjO+6+MCw/Bvhadf8a0AboCwwHHnT3BLDczP6zi/MPBaZWn8vddzcn3reA/sEjqQC0DmcWGQ58J/zsM2a2PoWf6QIzOyVc7xbGuhaoAh4Oy+8DHg+vcRjwaI1rN0vhGtJIKcHllopwCp6dwn/om2sWAb909ylfOq4+nxXNA4a6+9ZdxJIyMxtBkCwPdfctZvYKULibwz287oYvfwciu6M+uPiZAvw8nM4IM9vXzIqAqcD3wz66EuDIXXz2LWC4mfUKP9suLN8EtKpx3PPAL6s3zGxQuDoVOCMsOw7Yq5ZY2wDrw+S2H0ENsloeUF0LPYOg6bsRWGhm3wuvYWY2sJZrSCOmBBc/dxD0r82w4KU5/yKoqT8BfBLuu4dgtpQvcPc1wFiC5uAsPm8iPgWcUj3IAFwADAkHMebw+WjuHwgS5IcETdXPaol1MtDEzOYCfyFIsNU2E0wUOZugj+2PYfkPgHPD+D4ERqfwnUgjpdlERCS2VIMTkdhSghOR2FKCE5HYUoITkdhSghOR2FKCE5HYUoITkdj6/x6LukwT98T1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Results for highest accuracy model\n",
      "Trained EEGNet on single session using train/test split and got accuracy of: 0.9197916666666667\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhLklEQVR4nO3dd5gV9dn/8fe9hQWWXqWsFEERMaJBRU2I5VHsJZcxlscWI7bYgknUJ9bE/JLHFmNLsPzE3tCIDSwxQaOoiIgIEpr0trD0BXbP3s8fM4urwtlzdvfsnDP7eXHNxcx35szcey6491tmvmPujohIHOVFHYCISKYowYlIbCnBiUhsKcGJSGwpwYlIbBVEHUBNnTrke++SwqjDyFr/mdoy6hCynuXpd3Yy5VUb2OqbrT7nGH5Isa9anUjp2E+mbhnv7kfW53r1kVUJrndJIR+NL4k6jKw1vPvgqEPIenkti6MOIatN3PRKvc9RujrBh+N7pnRsYbc5nep9wXrQrzsRSZOT8KqUlmTMrLmZfWRmn5nZF2Z2U1jex8w+NLPZZvaMmTULy4vC7dnh/t61RaoEJyJpcaAKT2mpxRbgUHffCxgMHGlmQ4E/AXe6ez+gDDgvPP48oCwsvzM8LiklOBFJW1WKf5LxwIZwszBcHDgUeD4sHw2cGK6fEG4T7j/MzJL2J2ZVH5yIZD/Hqail+VlDJzObVGN7lLuPqt4ws3zgE6AfcC8wB1jj7pXhIYuAHuF6D2AhgLtXmtlaoCNQuqOLK8GJSFocSNTe/KxW6u5Ddngu9wQw2MzaAS8CA+odYA1qoopI2hqoD24bd18DvAMcALQzs+rKV09gcbi+GCgBCPe3BVYlO68SnIikxYGEe0pLMmbWOay5YWYtgMOBGQSJ7uTwsLOBl8L1seE24f5/eC3TIamJKiJpS7kHLrluwOiwHy4PeNbdXzGz6cDTZvZ74FPgofD4h4DHzGw2sBo4tbYLKMGJSFocT6cPbsfncZ8K7L2d8rnAftsp3wz8JJ1rKMGJSFrcoSJH5slVghORNBkJ6vU4a6NRghORtDhQpRqciMSVanAiEkvBjb5KcCISQw5UeG7cQqsEJyJpcYxEjjwjoAQnImmrcjVRRSSG1AcnIjFmJNQHJyJxFMzoqwQnIjHkbmz1/KjDSIkSnIikrUp9cCISR8Egg5qoIhJLGmQQkZjSIIOIxFpCN/qKSBw5RoXnRurIjShFJGtokEFEYssxNVFFJL40yJDltm42Rv64HxVb80hUwg+PWctZv1rGsgXN+MNFvVhXVkD/PTfx67sXUNgsmJ/5X2Pb8fjtO4E5fQdu5pr75kf8U0SjsKiK21+YTWEzJ7/AeffVdjx2205RhxW5K//fbPY7ZDVrVhVy0THBy6LOvGIBBxy2miqHtasKuf03/Vm9olnEkdaPO7pNxMweBo4FVrj7oExdp64Ki5z/fW4OLYqrqKyAX57Yn30PXceYUZ358fkrOfjENdz1m56Me6oDx529isVzm/HM3V2446VZtG6XYE1pk/3dQMUW49c/2YXNm/LJL3Du+PtsPv5Ha76cXBx1aJF684XOjH1sJ666dda2sjEPduexP+8MwPFnLeX0Xyzknut3iSrEBhEMMuTGo1qZTMOPAEdm8Pz1YgYtioPX11ZWGIkKwww+e681Pzx2DQCH/2Q1H4xrC8DrT3TkuHNKad0uAUC7TpWRxJ0djM2bgn/gBYVOfqFTy0vMm4RpH7dl/dpv/uLbtOHr7eYtEjTA60SzQoK8lJaoZawa4u4TzKx3ps7fEBIJ+MXw3VjyVTOOO6eUbr22UNw2QX74rXTqVkHpskIAFs1tDsCVx/ejqsr475HL2PeQ9VGFHrm8POee8f+he++tvPxIR2Z+2rRrb8mcfeV8DjtpJRvX53P1mVnXmEmbYzkz4WX0KTZC+flw/1szeeKT6cyc0pKFs5vv8NhEAhbPK+LWMbO55r75/PmqEjaszY1qeiZUVRkXH74bZ3x/ILsN3kSv3cqjDilrjb6zF2cNG8I7Yztz3H8vjTqcBpErNbjIIzCzEWY2ycwmrVyViCSGVm0T7HXgBmZ80pKNa/NJhK3P0qWFdNqpAghqc0OPWEdBIey081Z67rKFxfNyu7O4IWxcl89n77dq0rXZVL0ztjMHDV8VdRj1FrwXNS+lJRkzKzGzd8xsupl9YWaXh+U3mtliM5sSLkfX+Mw1ZjbbzGaa2fDaYo08wbn7KHcf4u5DOndsvBrRmlX522pgW8qNyRNaU9J/C3sdtIF3X2kHwJvPdeCA4WsBOPDItUz9oBUAa1fls2hOEd123tpo8WaTth0qKW4T/DJq1ryKfYZtSFr7bcq69/q6ZnvAf61m0dwWEUbTUII326ey1KISGOnuA4GhwCVmNjDcd6e7Dw6X1wDCfacCexD0799nZkmTRpMdCly9vJDbLt+ZqiqjqgqGHbeGoYevo9eum/nDRb145H+70W9QOcNPWw3AkIPXM/lfrTn/RwPIy3fOv24JbTpEU+OMWoeuFVx11wLy8iAvDya83JYP32oTdViR+82d/+F7+62lTftKHnt3Eo/dVcK+B5fRs085XmWsWFLE3df3jTrMegteG1j/yoi7LwWWhuvrzWwG0CPJR04Annb3LcA8M5sN7Ad8sKMPmGdo+MvMngIOBjoBy4Eb3P2hZJ8Zsldz/2h8SUbiiYPh3QdHHULWyyvWYEcyEze9wtpEab1GCHrs0c4vfvYHKR3720GvfuLuQ2o7LhyQnAAMAn4JnAOsAyYR1PLKzOweYKK7Px5+5iHgdXd/fkfnzVgT1d1Pc/du7l7o7j1rS24ikjsSnpfSAnSq7mMPlxHfPpeZtQLGAFe4+zrgfmAXYDBBDe/2usbZZJuoIlI3wXxwKVcCS5PV4MyskCC5PeHuLwC4+/Ia+x8AXgk3FwM1m3g9w7IdinyQQURyjaVTg9vxWcwMeAiY4e531CjvVuOwk4Bp4fpY4FQzKzKzPkB/4KNk11ANTkTSEtwm0iA3+h4EnAl8bmZTwrJrgdPMbHB4qa+ACwDc/QszexaYTjACe4m7Jx3pU4ITkbQ01LOo7v4ebLet+1qSz9wC3JLqNZTgRCRtmi5JRGIpmC4pN55FVYITkbTlysP2SnAikpZgNhE1UUUkhoJHtZTgRCSWVIMTkRhL40mGSCnBiUhaNIoqIrGmJqqIxFIuvZNBCU5E0uJApWpwIhJXaqKKSDy5mqgiElNpTngZKSU4EUmbanAiEksNOOFlxinBiUhaHKOySoMMIhJT6oMTkXhyNVFFJKbUBycisaYEJyKx5BgJDTKISFxpkEFEYsk1yCAiceZKcCIST3rYXkRiTDW4OvjP1JYM7/n9qMPIWmMW/TvqELLeybv8KOoQspq7N8A5IFFV/wRnZiXAo0BXgtvrRrn7XWbWAXgG6A18BZzi7mVmZsBdwNHAJuAcd5+c7Bq5MdYrIlmlCktpqUUlMNLdBwJDgUvMbCBwNfC2u/cH3g63AY4C+ofLCOD+2i6gBCciaXGCJmoqS9LzuC+troG5+3pgBtADOAEYHR42GjgxXD8BeNQDE4F2ZtYt2TWyqokqIrkgrUGGTmY2qcb2KHcf9Z0zmvUG9gY+BLq6+9Jw1zKCJiwEyW9hjY8tCsuWsgNKcCKStjS68krdfUiyA8ysFTAGuMLd1wVdbdXXcTezOnccqokqImlriCYqgJkVEiS3J9z9hbB4eXXTM/x7RVi+GCip8fGeYdkOKcGJSFqCUdS8lJZkwlHRh4AZ7n5HjV1jgbPD9bOBl2qUn2WBocDaGk3Z7VITVUTS1gB3mwAcBJwJfG5mU8Kya4E/As+a2XnAfOCUcN9rBLeIzCa4TeTc2i6gBCciaWuIG33d/T3Y4b0kh23neAcuSecaSnAikhYntf61bKAEJyJpa5gWauYpwYlIehy8AR7VagxKcCKSNjVRRSS2GmgUNeN2mODM7G6SNLXd/bKMRCQiWa36WdRckKwGNynJPhFpqhzI9QTn7qNrbptZS3fflPmQRCTb5UoTtdZHtczsADObDnwZbu9lZvdlPDIRyVKGV6W2RC2VZ1H/DAwHVgG4+2fAsAzGJCLZzlNcIpbSKKq7L6w5hQmQyEw4IpL1PB6DDNUWmtmBgIdTm1xOMPOmiDRVWVA7S0UqTdQLCR5w7QEsAQaT5gOvIhI3luISrVprcO5eCpzRCLGISK6oijqA1KQyitrXzF42s5VmtsLMXjKzvo0RnIhkoer74FJZIpZKE/VJ4FmgG9AdeA54KpNBiUh2c09tiVoqCa6luz/m7pXh8jjQPNOBiUgWy/XbRMK3SwO8bmZXA08ThPxTgqmDRaSpyoLmZyqSDTJ8QpDQqn+SC2rsc+CaTAUlItmt7i/ya1zJnkXt05iBiEiOcIMseAwrFSk9yWBmg4CB1Oh7c/dHMxWUiGS5XK/BVTOzG4CDCRLca8BRwHuAEpxIU5UjCS6VUdSTCV7htczdzwX2AtpmNCoRyW65PopaQ7m7V5lZpZm1AVYAJRmOK1LFbSq58tYF9N6tHHe4Y2QvZkxuFXVYjap0STP+cvkurC0tBIPDT1/BsT9fxlfTW/K3q/uweWM+nUu2cMXds2nZOkHFVuNvV/dhzmetsDznZzfNZ9CB66L+MSJzwjnLOOrUlZjB60935u//f6eoQ2o4cZjwsoZJZtYOeIBgZHUD8EFtHzKzEoJmbFeCr2SUu99V91Abz0U3LWLSP9vw+wv6UlBYRVGLHHkupQHl5zvnXD+fvntuonxDHr86ak/2GraW+37Vl7N/O589DljP20935qW/duO0Xy3irSe7AHDn21NZW1rA788cwJ9enUZeKm2EmOm16yaOOnUll584kIqKPG55ZCYf/qMdS+fH5/bRXBlFrfWfn7tf7O5r3P2vwOHA2WFTtTaVwEh3HwgMBS4xs4H1CzfzWrZOsOf+Gxj3VEcAKivy2Liu6b2bp33XCvruGUzg3KJVFT37l7N6WTOWzm3OwKHrAdhr2FomvhbcLrloVottNba2nSopbpNgzmfF0QQfsZ37bWbmlGK2bM6nKmF8/lFrDjqyLOqwGlaONFF3mODMbJ9vL0AHoCBcT8rdl7r75HB9PcEUSz0aKvBM2alkC2tXFzDyjvncO24GV9w6n6IWTXv6uxULi5g3rZj+e2+gZNdyPhrfHoD3X+lA6ZIiAHrtvolJb7YnUQnLFxQx5/Pibfuamq9mtmCP/dbTul0FRc0T7HvwGjp32xJ1WA3KPLUlasmqJrcn2efAoalexMx6A3sDH25n3whgBEBzWqZ6yozJL3D6DdrEvdeVMPPTYi68aSE/vWQ5j97WPerQIlG+MY9bR/Tn3Bu/omXrBBffPoeHr+/N83f1YN/DyygoDJrvh526gsWzW/Dro/ekc88t7Pb99eTlZ8G/8AgsnNOC5/7anT88OpPN5fnMmV5MVSI3+qxS1kB9cGb2MHAssMLdB4VlNwLnAyvDw65199fCfdcA5xFMunuZu49Pdv5kN/oeUu/og4BaAWOAK9z9O73O7j4KGAXQxjpE/j+idGkzVi5txsxPg+bVe6+255RLlkUcVTQqK4xbR+zKD08qZejRQROrZ7/NXP/klwAsmducT94OanP5BXDujfO3ffbaE/age9/NjR90lhj/bGfGP9sZgHOuWkjpsmYRR9SAGrb5+QhwD9+97exOd7+tZkHYxXUqsAfBxB9vmdmu7r7DJlZGu4DDGYDHAE+4+wuZvFZDKVtZSOmSQnqG/zkH/2AdC2bFp3M4Ve5w31V96dmvnONHfJ3g15YGvxOrquD5u3pwxJnLAdhSnsfmTcE/p88mtCWvwCnZtbzxA88SbTtWANC5+xYOOrKMd17qGHFEDayB+uDcfQKwOsWrngA87e5b3H0eMBvYL9kHMtZ7bsFLHB4CZrj7HZm6Tibce10Jv7n7KwqaVbFsfhG3j+wVdUiN7suPW/OvMZ3ZecBGRh6xJwCn/2YhS+c1Z9zorgDsf9RqDv1p0IpYW1rI784YgOVBh522ctldsyOLPRtcd/8sWrerJFFp3Ht9Lzauj9dAlaV+Y0EnM6v5juVRYautNr8ws7MI3s880t3LCPrwJ9Y4ZhG19Otn8ls/CDgT+NzMpoRl29rS2Wzu9JZcesyAqMOI1O77rWfMoonb3Xfsz7/bZO9SsoW7J3yW6bByxlWnZP0NA/WTehO11N2HpHn2+4HfhVf5HcF4wM/SPAeQ2qNaRjBleV93v9nMdgZ2cvePkn3O3d8jGyZlF5EGlekRUndfvu1aZg8Ar4Sbi/nmQwY9w7IdSqUP7j7gAOC0cHs9cG+qwYpIDGVwynIz61Zj8yRgWrg+FjjVzIrMrA/QH0ha0Uqlibq/u+9jZp8CuHuZmcVoSEhE0tZANTgze4pgMo9OZrYIuAE42MwGh1f5inAuSnf/wsyeBaYTPEhwSbIRVEgtwVWYWX54McysMznzTh0RyYSGaqK6+2nbKX4oyfG3ALekev5UEtxfgBeBLmZ2C8HsIr9N9QIiEjOe1ihqpFJ5L+oTZvYJwZRJBpzo7nqzvUhTFvkt+alJZRR1Z2AT8HLNMndfkMnARCSLxSXBAa/y9ctnmgN9gJkEj0uISBOUDQ/SpyKVJuqeNbfDmUQuzlhEIiINJO0nGdx9spntn4lgRCRHxKUGZ2a/rLGZB+wDLMlYRCKS3eI0igq0rrFeSdAnNyYz4YhITohDDS68wbe1u1/VSPGISJYzYjDIYGYF7l5pZgc1ZkAikgNyPcERPMS6DzDFzMYCzwEbq3fmygSWItLAsuR9C6lIpQ+uObCK4B0M1ffDOaAEJ9JUxWCQoUs4gjqNrxNbtRzJ3yKSCXGoweUDrdj+pJU58uOJSEbkSAZIluCWuvvNjRaJiOSGLHmpcyqSJThNNy4i2xWHJuphjRaFiOSWXE9w7p7quwpFpImJ06NaIiJfi0kfnIjIdxi500GvBCci6VMNTkTiKg6jqCIi26cEJyKxFLMJL0VEvkk1OBGJK/XBiUh8KcGlzwoKyO/QIeowstZP9z0x6hCy3nUzXos6hKx2/vEbGuQ8DVWDM7OHgWOBFe4+KCzrADwD9Aa+Ak5x9zIzM+Au4GiCl9Gf4+6Tk50/r2HCFJEmwwkmvExlqd0jwJHfKrsaeNvd+wNvh9sARwH9w2UEcH9tJ1eCE5G0VL90JpWlNu4+Afj2c+8nAKPD9dHAiTXKH/XARKCdmXVLdn4lOBFJn6e4QCczm1RjGZHC2bu6+9JwfRnQNVzvASyscdyisGyHsqoPTkRyg3nKnXCl7j6krtdxdzere4+fanAikp5Ua291H4hYXt30DP9eEZYvBkpqHNczLNshJTgRSVtD9cHtwFjg7HD9bOClGuVnWWAosLZGU3a71EQVkbQ11KNaZvYUcDBBX90i4Abgj8CzZnYeMB84JTz8NYJbRGYT3CZybm3nV4ITkfQ10H1w7n7aDnZ955UJ7u7AJemcXwlORNITszfbi4h8kxKciMRR9Y2+uUAJTkTSZlW5keGU4EQkPXqrlojEmWb0FZH4Ug1OROJKgwwiEk8OpP6wfaSU4EQkbeqDE5FY0n1wIhJf7mqiikh8qQYnIvGlBCcicaUanIjEkwOJ3MhwSnAikjbV4EQkvjSKKiJxpRqciMSTpksSkbgywDTIICJxlcab7SOlBCci6VETNbd06rqZkbd8QfsOW3Fg3PM9eOnJnfnB4cs546K5lPTZyJVn7Mes6W2iDjUynbqWM/Lmz2nXYSvuMO7FEsY+1YvTR8xm+EmLWFfWDIDR9/Zn0r87Rxxt41i7pJC/X9WbjaUFmME+p5ay/7kref7SPqyaWwTA5nX5NG+T4IJXvyRRAS9f04tl01pSlTC+d9IqfnDx8oh/irrQs6iYWXNgAlAUXud5d78hU9erj0TCePC2/sz5sg0tWlbyl6c/YvLEDsyf3YrfX/k9Lr1uRtQhRi6RyOPBOwds+47uevwDPp3YEYCXnuzFC4/1iTjCxpdX4Bxx7SK6DSpny4Y8Hjh+AH1/sJ6T75637Zg3bulBUesEANNfa09iq3HhuBlUlBv3HTGQQceX0a7n1qh+hDrTKCpsAQ519w1mVgi8Z2avu/vEDF6zTspKiygrDX7jlm8qYMHclnTqsmXbf2D57ne0cF4xHbtsjjiqaLXuUknrLpUAFLWqolO/zaxbVkjn/sH34h4ktTMfnwWAGWzdlE9VJVRsziO/0ClqlYgs/npp6jU4d3dgQ7hZGC5Z/6106V7OLgPW8+XnbaMOJWt16VZO3wHrmTmtHQP3WsOxpyzg0GOWMGt6Wx66czc2rC+MOsRGt2ZRM5Z90ZKegzduK1vwcSuKO1bQsc8WAHY/qoyZb7XljqF7UlGexxG/XUSLdjmY4Dx3RlHzMnlyM8s3synACuBNd/8wk9err+YtKvmf26cy6tbdKN+o7sntad6ikv+5dQoP3DaA8o0FvPZ8CT8/YRiXnnYgZaVFnHflzKhDbHRbN+bx3MV9GX7dIopafz3V7bSx7Rl0fNm27cWfFWN5cOUHn3PZv75g4oNdKVvQLIqQ689TXGphZl+Z2edmNsXMJoVlHczsTTObFf7dvq5hZjTBuXvC3QcDPYH9zGzQt48xsxFmNsnMJm2tKs9kOEnlF1TxP3dM5Z+v7cT7b3eJLI5sll9QxbW3TuGd17vx/jtdAVizuoiqKsPdGPdiT3bdY23EUTauRAU8e3FfBh2/mt2PXLOtvKoSvhzfjj2O+TrBTRvbgX4/Wkd+IRR3qqTk+xtY8nnLCKKuP3NPaUnRIe4+2N2HhNtXA2+7e3/g7XC7TjKa4Kq5+xrgHeDI7ewb5e5D3H1Is7wWjRHOdjhX3DidhXOLefGxXhHFkO2cy6/7goXzivn7E723lbbvtGXb+oGHLGf+nFYRxBYNd3j56l503mUzB/x8xTf2zf13Gzruspk23Sq2lbXtvpV577cGYOumPBZNKaZT3y3kpOpZfWtb6uYEYHS4Pho4sa4nyuQoamegwt3XmFkL4HDgT5m6Xn0M3Hsthx23jHn/acXdzwRjIKPv7kdhsyouunombdtv5cZ7pjB3Ziuuu2ifiKONxsDBazjs2CXMm9WKu598HwhuCfnR8KX03W097rBiSQvu/sMeEUfaeBZOKmbqix3psls5fztmAACHXrWE/oes44tX2jPouLJvHL/vmSt56de9uH/47rjD4JNX0XX36FotdeZA6i+d6VTd9AyNcvdR3zrbG2bmwN/CfV3dfWm4fxnQta6hmmdoNMTMvkeQffMJaorPuvvNyT7TtrCLH9Dh5IzEEwdWkB91CFnv2n+/FnUIWe384xfx5dQtVp9ztC3u7kMHXpDSsW9MuvGTGk3P7zCzHu6+2My6AG8ClwJj3b1djWPK3L1O/XCZHEWdCuydqfOLSISqGua9ge6+OPx7hZm9COwHLDezbu6+1My6EQxS1kmj9MGJSIxUN1FTWZIws2Iza129DhwBTAPGAmeHh50NvFTXUHUvhIikrYEetu8KvGhmEOSiJ919nJl9DDxrZucB84FT6noBJTgRSV8DJDh3nwvstZ3yVcBh9b4ASnAikjY9bC8icaW3aolInGnCSxGJLyU4EYklB6qU4EQkljTIICJxpgQnIrHkQKJhHtXKNCU4EUmTgyvBiUhcqYkqIrGkUVQRiTXV4EQktpTgRCSW3CGRG687VIITkfSpBicisaUEJyLx5BpFFZGYcnDd6CsisaVHtUQkltwb7LWBmaYEJyLp0yCDiMSVqwYnIvGkCS9FJK70sL2IxJUDrke1RCSWXBNeikiMuZqoIhJbOVKDM8+i0RAzWwnMjzqOGjoBpVEHkcX0/dQu276jXu7euT4nMLNxBD9XKkrd/cj6XK8+sirBZRszm+TuQ6KOI1vp+6mdvqNo5UUdgIhIpijBiUhsKcElNyrqALKcvp/a6TuKkPrgRCS2VIMTkdhSghOR2FKC2w4ze9jMVpjZtKhjyUZmVmJm75jZdDP7wswujzqmbGJmzc3sIzP7LPx+boo6pqZKfXDbYWbDgA3Ao+4+KOp4so2ZdQO6uftkM2sNfAKc6O7TIw4tK5iZAcXuvsHMCoH3gMvdfWLEoTU5qsFth7tPAFZHHUe2cvel7j45XF8PzAB6RBtV9vDAhnCzMFxUk4iAEpzUi5n1BvYGPow4lKxiZvlmNgVYAbzp7vp+IqAEJ3VmZq2AMcAV7r4u6niyibsn3H0w0BPYz8zU1REBJTipk7BvaQzwhLu/EHU82crd1wDvAJE9cN6UKcFJ2sJO9IeAGe5+R9TxZBsz62xm7cL1FsDhwJeRBtVEKcFth5k9BXwA7GZmi8zsvKhjyjIHAWcCh5rZlHA5Ouqgskg34B0zmwp8TNAH90rEMTVJuk1ERGJLNTgRiS0lOBGJLSU4EYktJTgRiS0lOBGJLSW4HGJmifCWjGlm9pyZtazHuR4xs5PD9QfNbGCSYw82swPrcI2vzOw7b1/aUfm3jtmQbP92jr/RzK5KN0aJNyW43FLu7oPDGU62AhfW3GlmdXrPrbv/vJaZQA4G0k5wIlFTgstd7wL9wtrVu2Y2FpgePuR9q5l9bGZTzewCCJ4+MLN7zGymmb0FdKk+kZn908yGhOtHmtnkcC6zt8OH6S8Ergxrjz8M79QfE17jYzM7KPxsRzN7I5wD7UHAavshzOzvZvZJ+JkR39p3Z1j+tpl1Dst2MbNx4WfeNbMBDfJtSizpzfY5KKypHQWMC4v2AQa5+7wwSax1933NrAj4t5m9QTDjx27AQKArMB14+Fvn7Qw8AAwLz9XB3Veb2V+BDe5+W3jck8Cd7v6eme0MjAd2B24A3nP3m83sGCCVJ0B+Fl6jBfCxmY1x91VAMTDJ3a80s+vDc/+C4CUuF7r7LDPbH7gPOLQOX6M0AUpwuaVFOAUPBDW4hwiajh+5+7yw/Ajge9X9a0BboD8wDHjK3RPAEjP7x3bOPxSYUH0ud9/RnHj/BQwMHkkFoE04s8gw4MfhZ181s7IUfqbLzOykcL0kjHUVUAU8E5Y/DrwQXuNA4Lka1y5K4RrSRCnB5ZbycAqebcL/6BtrFgGXuvv4bx3XkM+K5gFD3X3zdmJJmZkdTJAsD3D3TWb2T6D5Dg738Lprvv0diOyI+uDiZzxwUTidEWa2q5kVAxOAn4Z9dN2AQ7bz2YnAMDPrE362Q1i+Hmhd47g3gEurN8xscLg6ATg9LDsKaF9LrG2BsjC5DSCoQVbLA6proacTNH3XAfPM7CfhNczM9qrlGtKEKcHFz4ME/WuTLXhpzt8IauovArPCfY8SzJbyDe6+EhhB0Bz8jK+biC8DJ1UPMgCXAUPCQYzpfD2aexNBgvyCoKm6oJZYxwEFZjYD+CNBgq22kWCiyGkEfWw3h+VnAOeF8X0BnJDCdyJNlGYTEZHYUg1ORGJLCU5EYksJTkRiSwlORGJLCU5EYksJTkRiSwlORGLr/wAUpeSdLoxjvgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################################\n",
    "# MULTI TRIAL EEGNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Configure\n",
    "retrain_model = False\n",
    "tensorboard_name = \"EEGNet_multisession_C0_100hz\"\n",
    "best_model_filename = f\"./saved_variables/6/EEGNet/{tensorboard_name}\"\n",
    "\n",
    "\n",
    "# Train on train/test split of data from one session\n",
    "## Note: the model is forced to use GPU, if GPU is not available replace with what is available e.g. /cpu:0\n",
    "if (retrain_model): # Retrain or not\n",
    "    with tf.device('/gpu:0'):\n",
    "        keras_eegnet_model.fit(\n",
    "            x= mne_fixed_window_epochs_train_data,\n",
    "            y= ohe_labels_train,\n",
    "            batch_size= 128, # Default: 32\n",
    "            epochs= 500, # Default: 500 (EEGNet paper)\n",
    "            verbose= 1, # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "            #callbacks= [tensorboard_callback(\"EEGNet_raw_signal_0.5s_100samps_50kernlen_02nr\")], # To be used for TF Board\n",
    "            callbacks= [tensorboard_callback(tensorboard_name),\n",
    "                        lowest_loss_model_save_callback(best_model_filename),\n",
    "                        highest_accuracy_model_save_callback(best_model_filename)],\n",
    "            validation_split= 0.3,\n",
    "            shuffle= True,\n",
    "            sample_weight= None, # Can be interesting due to time series\n",
    "            use_multiprocessing=True, # Done for faster speed\n",
    "            workers= 4 # Done for faster speed\n",
    "            )\n",
    "\n",
    "# Convert labels back to original\n",
    "y_test = ohe.inverse_transform(ohe_labels_test)\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"Results for lowest loss model\")\n",
    "keras_eegnet_model = load_lowest_loss_model(best_model_filename)\n",
    "\n",
    "y_pred = keras_eegnet_model.predict(mne_fixed_window_epochs_test_data)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained EEGNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"\\n\\nResults for highest accuracy model\")\n",
    "keras_eegnet_model = load_highest_accuracy_model(best_model_filename)\n",
    "\n",
    "y_pred = keras_eegnet_model.predict(mne_fixed_window_epochs_test_data)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained EEGNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Remove unused variables\n",
    "del accuracy\n",
    "del best_model_filename\n",
    "del retrain_model\n",
    "del tensorboard_name\n",
    "del y_pred\n",
    "del y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bd23746d",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAUP\n",
    "####################################################\n",
    "\n",
    "# delete unused variables\n",
    "del keras_eegnet_model\n",
    "del mne_fixed_window_epochs_test_data\n",
    "del mne_fixed_window_epochs_train_data\n",
    "del ohe\n",
    "del ohe_labels_test\n",
    "del ohe_labels_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28989ef2",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## ShallowConvNet\n",
    "\n",
    "The ShallowConvNet and DeepConvNet are implementations of the models described by Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730)).\n",
    "They are shallow and deep convolutional neural networks both presented in the same paper.\n",
    "CNN's have been proven to be very good at image classification tasks, thus EEG-as-image approaches have been used in the past where an image representation of the EEG signal is processed by popular CNN architectures.\n",
    "The approach by Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730) works with raw EEG however, which seems like a better approach.\n",
    "This makes the pipeline more efficient but also means that the architectures from image processing can't be blindly adopted.\n",
    "Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730)) have shown that their method outperforms Filter Bank Common Spatial Patterns (FBCSP), so it should outperform our naive CSP experiment results from the previous notebooks.\n",
    "\n",
    "We will first explore the ShallowConvNet method.\n",
    "The original implementation of the model works on 250Hz data.\n",
    "The defaults provided by the EEGModels library are adopted for 128Hz, we will further adapt it to 100Hz here.\n",
    "Contrary to the EEGNet implementation of EEGModels, the implementation of ShallowConvNet and DeepConvNet are not verified by the original authors, Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730)).\n",
    "However, the found results are in line with the paper, thus the implementation is believed to be correct.\n",
    "\n",
    "### Fixed window classification: Single trial | ShallowConvNet | three class MI task | 100Hz input signal\n",
    "\n",
    "As discussed, the original implementation of the model works on 250Hz data.\n",
    "The defaults provided by the EEGModels library are adopted for 128Hz, we will further adapt it to 100Hz here.\n",
    "If you remember from the previous notebooks, the data was sampled at 200Hz, which means there are 200 samples per second.\n",
    "In the previous notebook we processed data of 0.5 seconds, thus 100 samples.\n",
    "To get initial results, we will use the model on this 100 samples data.\n",
    "\n",
    "This more shallow architecture is inspired by FBCSP and specifically tailored to decode band power features as discussed by Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730).\n",
    "Because of this, it is expected to be far better then CSP and ideally it would rival EEGNet on single trial performance.\n",
    "However, as it is still based on CSP, albeit a FBCSP, it is expected that problems could arise with inter-session and inter-patient experiments.\n",
    "\n",
    "\n",
    "The scores are given for the best CSP approach as well as for the trained DL approach.\n",
    "Scores for the DL approach are a combination of the best model based on validation loss / validation accuracy.\n",
    "TensorBoard was used to monitor the behaviour of the model over time, to ensure no overfitting tendencies occur and that enough epochs have happened to learn optimally.\n",
    "\n",
    "**The result on subject `C` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** |\n",
    "|----------------|------------------------------|---------------------------|-----------------------------------|\n",
    "| 0              | 0.896                        | 0.944 / 0.934             | 0.951 / 0.955                     |\n",
    "| 1              | 0.847                        | 0.931 / 0.913             | 0.937 / 0.937                     |\n",
    "| 2              | 0.719                        | 0.892 / 0.878             | 0.868 / 0.871                     |\n",
    "\n",
    "**The result on subject `B` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** |\n",
    "|----------------|------------------------------|---------------------------|-----------------------------------|\n",
    "| 0              | 0.465                        | 0.569 / 0.587             | 0.496 / 0.542                     |\n",
    "| 1              | 0.560                        | 0.701 / 0.684             | 0.649 / 0.601                     |\n",
    "| 2              | 0.653                        | 0.816 / 0.792             | 0.694 / 0.712                     |\n",
    "\n",
    "\n",
    "**The result on subject `E` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** |\n",
    "|----------------|------------------------------|---------------------------|-----------------------------------|\n",
    "| 0              | 0.701                        | 0.792 / 0.785             | 0.802 / 0.806                     |\n",
    "| 1              | 0.580                        | 0.795 / 0.799             | 0.826 / 0.823                     |\n",
    "| 2              | 0.805                        | 0.864 / 0.864             | 0.916 / 0.923                     |\n",
    "\n",
    "The training of this network is considerably faster as it takes less epochs to converge.\n",
    "The modification to let us set extra parameters for this model has shown to be improving of the model's performance on all aspects.\n",
    "However, with a near 1 accuracy, there are signs of overfitting.\n",
    "Being a somewhat CSP based approach, it is expected that well MI capable patient performance would be high whilst hard patients such as subject `B` would be low."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f134c1dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 0 ... 666799  =      0.000 ...  3333.995 secs...\n",
      "Using data from preloaded Raw for 960 events and 801 original time points ...\n",
      "0 bad epochs dropped\n",
      "Loaded fixed window binary epochs:\n",
      "\n",
      "Extracted labels from epochs: [1 2 1 3 1 3 3 2 3 2]\n",
      "One Hot Encoded labels: [[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]]\n",
      "Labels match before and after the One Hot Encoding: True\n",
      "Shape of data (epochs, channels, samples): (960, 21, 100)\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# PREPPING THE DATA\n",
    "####################################################\n",
    "\n",
    "# Specify raw to use\n",
    "mne_raw = load_mne_raw(subject= \"C\", index= 0)\n",
    "\n",
    "# Get the epoch from the RAW limited to MI tasks\n",
    "# Include period before and after to enable filtering possibilities\n",
    "mne_fixed_window_epochs = CLA_dataset.get_usefull_epochs_from_raw(mne_raw, start_offset=-1.5, end_offset=1.5)['task/neutral', 'task/left', 'task/right']\n",
    "\n",
    "# Load the epochs, we don't need filtering since this is what we want to learn!\n",
    "mne_fixed_window_epochs.load_data()\n",
    "print(f\"Loaded fixed window binary epochs:\\n\")\n",
    "\n",
    "# Labels: should be one hot encoded!\n",
    "labels = mne_fixed_window_epochs.events[:, -1]\n",
    "print(f\"Extracted labels from epochs: {labels[:10]}\")\n",
    "\n",
    "# Go to 2D representation\n",
    "labels = labels.reshape(-1, 1)\n",
    "\n",
    "# One Hot Encode the labels\n",
    "ohe = OneHotEncoder()\n",
    "ohe_labels = ohe.fit_transform(labels).toarray()\n",
    "print(f\"One Hot Encoded labels: {ohe_labels[:10]}\")\n",
    "\n",
    "# Show ohe labels\n",
    "np.shape(ohe_labels)\n",
    "ohe_labels[:10]\n",
    "\n",
    "# Validate OHE\n",
    "print(f\"Labels match before and after the One Hot Encoding: {np.array_equal(ohe.inverse_transform(ohe_labels), labels)}\")\n",
    "\n",
    "# Get effective data (half a second)\n",
    "mne_fixed_window_epochs_data = mne_fixed_window_epochs.get_data(tmin=0.2, tmax=0.7)\n",
    "\n",
    "# Fix scaling sensitivity as MNE stores as data * 10e-6\n",
    "mne_fixed_window_epochs_data = mne_fixed_window_epochs_data * 1000000\n",
    "\n",
    "# Delete unused variables\n",
    "del mne_fixed_window_epochs\n",
    "print(f\"Shape of data (epochs, channels, samples): {np.shape(mne_fixed_window_epochs_data)}\")\n",
    "\n",
    "# Remove unused variables\n",
    "del labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9142e52",
   "metadata": {},
   "source": [
    "#### Note on the ShallowConvNet model\n",
    "\n",
    "The ShallowConvNet implementation used was provided by the Army Research Laboratory on [GitHub](https://github.com/vlawhern/arl-eegmodels).\n",
    "However, they hard coded some model parameters we found important to modify and thus we edited their model definition slightly to allow for setting extra parameters dynamically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "034396c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_3 (InputLayer)        [(None, 21, 100, 1)]      0         \n",
      "                                                                 \n",
      " conv2d_2 (Conv2D)           (None, 21, 91, 40)        440       \n",
      "                                                                 \n",
      " conv2d_3 (Conv2D)           (None, 1, 91, 40)         33600     \n",
      "                                                                 \n",
      " batch_normalization_6 (Batc  (None, 1, 91, 40)        160       \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_4 (Activation)   (None, 1, 91, 40)         0         \n",
      "                                                                 \n",
      " average_pooling2d_4 (Averag  (None, 1, 11, 40)        0         \n",
      " ePooling2D)                                                     \n",
      "                                                                 \n",
      " activation_5 (Activation)   (None, 1, 11, 40)         0         \n",
      "                                                                 \n",
      " dropout (Dropout)           (None, 1, 11, 40)         0         \n",
      "                                                                 \n",
      " flatten (Flatten)           (None, 440)               0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 3)                 1323      \n",
      "                                                                 \n",
      " activation_6 (Activation)   (None, 3)                 0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 35,523\n",
      "Trainable params: 35,443\n",
      "Non-trainable params: 80\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE SHALLOWCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Create the TensorFlow Keras model\n",
    "keras_shallowcn_model = ShallowConvNet(\n",
    "    nb_classes = 3, # int, number of classes to classify. \n",
    "    Chans = 21, # number of channels in the EEG data. \n",
    "    Samples = 100, # number of time points in the EEG data. (default: 128, paper: 250)\n",
    "    dropoutRate = 0.8, # dropout fraction. (default: 0.5)\n",
    "    conv_filters = 10, # Conv2D kernel size (default: 13, paper: 25)\n",
    "    strides = 6, # Stride size for average pooling layer (default: 7, paper: 15)\n",
    "    pool_size = 30 # Pool size for average pooling layer (default: 35, paper: 75)\n",
    "    )\n",
    "\n",
    "# Compile the model so it can be fitted\n",
    "# Loss and optimizer from EEGNet paper\n",
    "keras_shallowcn_model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics=[\"accuracy\"])\n",
    "\n",
    "# Show summary of the model\n",
    "keras_shallowcn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1125216c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for lowest loss model\n",
      "Trained ShallowConvNet on single session using train/test split and got accuracy of: 0.9513888888888888\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAaxElEQVR4nO3deZhdVZnv8e+vhqQyhwxkIkCAmNyATE8MIE/TERFit91R27YFBG6LMjQool7FoS9qtwiKoAIKERRQBhmbQSFhvIhoBkIgE0NIJGYyM2QgQ1W994+zKxSQ1DknqVN771O/z/Psp87e+5y13zpP8tZaa6+1tiICM7M8q0k7ADOzPeVEZma550RmZrnnRGZmuedEZma5V5d2AK0N6Fcb+w+vTzuMzHp5do+0Q8g+34Vv0xY2sS22ak/KOOkDPWLN2qaS3vvsC1snR8SEPbleKTKVyPYfXs+0ycPTDiOzJow4Ku0QMi+2N6YdQqZNbZqyx2WsXtvE1Mn7lPTe+iGvDtjjC5YgU4nMzPIgaIrmtIN4GycyMytLAM1kqwnvRGZmZWvGNTIzy7Eg2O6mpZnlWQBNblqaWd65j8zMci2ApoyN13MiM7OyZauHzInMzMoUhPvIzCzfImB7tvKYE5mZlUs0sUfTNdudE5mZlSWAZtfIzCzvXCMzs1wrDIh1IjOzHAtge2RrTVYnMjMrSyCaMra4tBOZmZWtOdy0NLMccx+ZmVUB0eQ+MjPLs8IKsU5kZpZjEWJb1KYdxts4kZlZ2ZrdR2ZmeVbo7HfT0sxyzZ39ZpZz7uw3s6rQ5AGxZpZngdge2Uod2YrGzDIvi5392YrGzDIvEE1R2laMpAslzZU0R9JtkhokjZA0VdICSb+V1KVYOU5kZla2ZmpK2toiaRjwBWBsRBwC1AKfAi4DroyIg4B1wJnF4nEiS9x7/QDO+sAoPjd+FPf8YuDbzt117UBOGno4r6/J1mjmtAwYspXLbp3PdVNe4LrJs5n4v1ekHVLmfOny1/jtrBe47tF5aYfS7iKgKWpK2kpQB3STVAd0B5YDxwN3JedvAj5arJCKJTJJv5S0UtKcSl2jvfzlxQYeuqU/P/3dy1z76EtMfaQ3SxcVarMrl9Yz8//1Yu9h21KOMjuaG8UvvrcvZ594KF/8+Bj+6fS/se9Bb6YdVqZMubMf3/z0QWmHURGFzv7akjZggKQZrbazdpQTsRS4HFhMIYG9DjwLrI+IxuRtS4BhxWKqZI3sRmBCBctvN4tf6croIzbT0D2orYNDj9nIH3/fF4Drvj2MM7+1DGXrbnOq1q7qwoK5PQB4c1Mtf13Qjf6DnehbmzO1FxvWV28NvomakjZgdUSMbbVNailD0l7ARGAEMBTowW7mjIolsoh4ClhbqfLb0/6jtzBnWg/eWFvLls1i+uO9WbWsnmce7s2Awds58OAtaYeYWYOGbeXAMZt5aVbPtEOxDhKI5ihtK+IEYFFErIqI7cA9wLFA36SpCbAPsLRYQR5+Aew7ciuf/I+VfP3kA2no3swBB7/J9m3i9qsG8f3bXk07vMxq6N7Et37+Ctf9175s3li9tQ97t3YafrEYOFpSd+BN4IPADOAJ4BPA7cAZwH3FCkq9s1/SWS3t51VrmlKLY8Ipa7lm8sv86N4F9OzTxH6jtrBicRfOPWE0p48bw6rl9Zx30ijWrnTuB6ita+Y/f/4KT9zXnz9O7pd2ONaBCs+1rClpa7OciKkUOvVnArMp5KNJwNeAL0laAPQHbigWU+r/K5M28ySAsYc1pPbYz/Wr6+g7oJGVS+r54+/78JMHX+Fjn1294/zp48Zw1UMv0ad/esk2O4ILL1vE4gXduOeGIWkHYx2u/Z40HhEXAxe/4/BCYFw55aSeyLLiu5/dnw3r6qitD86/ZAk9+zhh7crBYzdywsfXsOjFblzzu8JN6Rt/uA/Tn+ybbmAZctHVizj0mA306dfIb6bP5tc/GsLk2wekHVa7KDwOLltdCRVLZJJuA8ZTuP26BLg4IopWEdNyxf8saPP8zdOqbzzQ7po7oxcTRpT1B7PTufT8EWmHUDERKtps7GgVS2QRcXKlyjazdHk9MjPLtcJ6ZNkaWOlEZmZl8gqxZpZzheEXrpGZWY61zLXMEicyMyub1+w3s1wrLOPjpqWZ5Zz7yMws1wqrX7hpaWY5Vpii5ERmZrnmGpmZVQGP7DezXPNdSzOrCm5amlmutazZnyVOZGZWlgAaXSMzs7xz09LM8q20R711KCcyMyuLF1Y0s6rgGpmZ5ZoXVjSz3AtEY7M7+80s59xHZmb5Fm5amlnOuY/MzKqCE5mZ5VogmtzZb2Z5585+M8u1cGe/mVWDcCIzs3zzpHEzqwKukbXh5Re6c9LQw9MOI7MeXPpM2iFk3keGj0s7hKoXAU3NTmRmlnO+a2lmuRZkr2mZrVFtZpYDhc7+UraiJUl9Jd0l6UVJ8yUdI6mfpEckvZL83KtYOU5kZla2iNK2EvwEeDgiRgOHAfOBi4DHImIk8Fiy3yYnMjMrW4RK2toiqQ9wHHBDoczYFhHrgYnATcnbbgI+Wiwe95GZWVkKdy1LrgMNkDSj1f6kiJiUvB4BrAJ+Jekw4FngAmBQRCxP3rMCGFTsIk5kZla2EpuNAKsjYuwuztUBRwKfj4ipkn7CO5qRERGSil7NTUszK1t7NC2BJcCSiJia7N9FIbH9TdIQgOTnymIFOZGZWVmC0pJYsUQWESuAv0oalRz6IDAPuB84Izl2BnBfsZjctDSzspXesizq88AtkroAC4F/p1DBukPSmcBrwCeLFeJEZmblCYh2mqIUEbOAnfWhfbCccpzIzKxsWRvZ70RmZmUr465lh9hlIpN0FW00hSPiCxWJyMwyLYtzLduqkc1o45yZdVYB5CWRRcRNrfcldY+IzZUPycyyLmtNy6LjyJLZ6POAF5P9wyT9rOKRmVlGiWgubesopQyI/TFwErAGICKepzDR08w6qyhx6yAl3bWMiL9Kb8uuTZUJx8wyL/LV2d/ir5LeD4Skegqz0+dXNiwzy7S89ZEB5wDnAcOAZcDhyb6ZdVoqcesYRWtkEbEaOLUDYjGzvGhOO4C3K+Wu5QGSHpC0StJKSfdJOqAjgjOzDGoZR1bK1kFKaVreCtwBDAGGAncCt1UyKDPLtnZcs79dlJLIukfEryOiMdl+AzRUOjAzy7C8DL+Q1C95+ZCki4DbKYT2b8DvOyA2M8uqHA2/eJZC4mqJ+OxW5wL4eqWCMrNsK76Kfsdqa67liI4MxMxyIgQdOP2oFCWN7Jd0CDCGVn1jEXFzpYIys4zLS42shaSLgfEUEtnvgQ8DTwNOZGadVcYSWSl3LT9BYf3sFRHx7xQea96nolGZWbbl5a5lK29GRLOkRkm9KTxjbniF40rV2PFvcM5/LaO2Jnjotn7ccXXRBx13CvddvzeTbx0AASedspqJn1vJr38wlKlT+iBB3wGNfPHKv9B/8Pa0Q03dly5/jaNOeJ31q+s4+4QxaYfTvjK4sGIpNbIZkvoCv6BwJ3Mm8KdiH5I0XNITkuZJmivpgj0LtWPU1ATnXbKUb506gs+NH8UHJq5n35Fb0g4rdX95sYHJtw7git/N56pH5jHt0T4sW9SVfzl3BVc/Op+rHpnP+05Yz21XDkk71EyYcmc/vvnpg9IOo2IUpW0dpWgii4j/iIj1EXEt8CHgjKSJWUwj8OWIGAMcDZwnKfN/mkYdsZllf+nCisVdadxew5P39eWYk15PO6zULXmlgVFHbKKhW1BbB4ccvYFnHupL915vTbrbsrkWZesPdWrmTO3FhvW1aYdROXlpWko6sq1zETGzrYIjYjmwPHm9QdJ8CitozNvNWDtE/8HbWbWsy4791cvrGX2kV/jeb/QWbr5sGG+sraVLt2ZmPN6HkYcVvpebLx3K43f1p3vvJr5/58spR2odITfjyIAftXEugONLvYik/YEjgKk7OXcWcBZAA91LLdI62PCRW/jEeSv4z1NG0tC9mQMOfpOamsK/5tMvWsbpFy3jjqsG8+CvBnLqV5anHK1VXMb6yNoaEPuB9riApJ7A3cAXI+KNnVxnEjAJoLf6pZ7n16yoZ+DQbTv2BwzZzurl9SlGlB0nnryGE09eA8BN3x/KgCFv79Qf//E1fPu0kU5k1a6Dm42lKKWzf7clK8reDdwSEfdU8lrt5aVZ3Rk2YhuDhm+lrr6Z8RPX8+cpHm0CsH514e/eyqX1/Omhvfj7j61l6cKuO85PndyXfQ70jZFOIS99ZHtKhUX+bwDmR8QVlbpOe2tuEtd8cxiX3LqQmlqYcns/XnvZi30AXPK5A9iwro7auuCc7y2mZ58mfvqV/VjyagM1NcHAYds479LFaYeZCRddvYhDj9lAn36N/Gb6bH79oyFMvn1A2mG1G2VsYcWKJTLgWOA0YLakWcmxb0RE5lfOmP54b6Y/3jvtMDLnB/e+uyP/G79YmEIk2Xfp+VU+VTljTctSpiiJwlLXB0TEdyXtCwyOiGltfS4inqYjF+02sw7R0WPESlFKH9nPgGOAk5P9DcA1FYvIzLIvY0tdl9K0PCoijpT0HEBErJPUpdiHzKyKZaxGVkoi2y6pliR0SQPJ3DNUzKwjZa1pWUoi+ylwL7C3pO9RWA3jWxWNysyyK3J41zIibpH0LIWlfAR8NCL8pHGzzixvNbLkLuVm4IHWxyLCA4bMOqu8JTLgd7z1EJIGYATwEnBwBeMyswzLWh9ZKcv4vDciDk1+jgTGUcJ6ZGZmpZBUK+k5SQ8m+yMkTZW0QNJvSxklUfZcy2T5nqN2I14zqxbtO9fyAqB1v/tlwJURcRCwDjizWAGl9JF9qdVuDXAksKzkEM2surTjXUtJ+wD/CHwP+FIyk+h44JTkLTcB3wZ+3lY5pfSR9Wr1upFCn9ndZcZrZtWk9NrWAEkzWu1PSpbuavFj4Ku8lWf6A+sjojHZX0JhQdY2tZnIkoGwvSLiK6VGbWbVTZTV2b86IsbutBzpI8DKiHhW0vg9iamtpa7rIqJR0rF7cgEzq0Ltc9fyWOCfJf0DhRERvYGfAH1b8g+wD7C0WEFtdfa3rG4xS9L9kk6T9PGWbQ9/ATPLqxKfoFSs1hYRX4+IfSJif+BTwOMRcSrwBIUZRABnAPcVC6mUPrIGYA2FDriW8WQB5GLFVzOrgMpOUfoacLuk/waeo7BAa5vaSmR7J3cs5/BWAmuRseFwZtaR2ntAbEQ8CTyZvF5IYbxqydpKZLVAT3a+OKITmVlnlrEM0FYiWx4R3+2wSMwsHzL4FKW2EpmXqTazncraXMu2EtkHOywKM8uXvCSyiFjbkYGYWX7kbmFFM7O3yVkfmZnZu4jsdaA7kZlZ+VwjM7O8y9NdSzOznXMiM7Ncy+Pj4MzM3sU1MjPLO/eRmVn+OZHtmmprqO3ZO+0wMmvikR9OO4TMu+zVB4q/qRP79D9tbJdyXCMzs3wLKr2wYtmcyMysLGU+fKRDOJGZWfmcyMws7xTZymROZGZWHq9+YWbVwH1kZpZ7nqJkZvnnGpmZ5VoJTxHvaE5kZlY+JzIzyzMPiDWzqqDmbGUyJzIzK4/HkZlZNfDwCzPLP9fIzCzv3NlvZvkWgCeNm1neuY/MzHLN48jMLP8i3LQ0s/xzjczM8i9jiawm7QDMLH8UpW1tliENl/SEpHmS5kq6IDneT9Ijkl5Jfu5VLB4nMjMrTwBNUdrWtkbgyxExBjgaOE/SGOAi4LGIGAk8luy3yYnMzMrWHjWyiFgeETOT1xuA+cAwYCJwU/K2m4CPFovHfWRmVr52vmspaX/gCGAqMCgilienVgCDin3eiczMylbGXcsBkma02p8UEZPeVpbUE7gb+GJEvCFpx7mICKn41ZzIzKw85S3jszoixu7qpKR6Cknsloi4Jzn8N0lDImK5pCHAymIXcR+ZmZVFgJqipK3NcgpVrxuA+RFxRatT9wNnJK/PAO4rFpNrZGZWtnZ60vixwGnAbEmzkmPfAC4F7pB0JvAa8MliBTmRmVl52mmF2Ih4mkIFb2c+WE5ZTmS7UFMT/OSu51izsivfPufgtMPJnB49t3PBxfPY78CNRIgff2cML77QN+2wUvWHGwYz7bcDkWDwqM386w8Xcv2nR7N1Uy0AG9fUM/ywjZwx6ZWUI91TnWiupaQG4Cmga3KduyLi4kpdr71NPH0pf13Yne49m9IOJZPO/upLPPtMfy75P4dRV9dM14bO/T29vqKeP944iC8/8gL1DcFvzjuI5x/oz7l3zt/xnl+fO5IxJ6xLMcr2k7W5lpXs7N8KHB8RhwGHAxMkHV3B67Wb/oO28r6/X8vkOwenHUomde+5nUOOXMfke4cB0NhYw6aN9SlHlb7mJrF9Sw1NjbB9Sw29996+49yWDbW8+kxvDj6xOhLZjhUwim0dpGI1sogIYGOyW59sGcvjO3f2N17ll5ePoFuPzl3L2JXBQ7fw+rouXPiduRzwno0smN+La38wmq1batMOLTV9Bm/nuM8t5/vHHkF9QzMj/+513nPc6zvOz52yFwe+/w0aelXBv6mg6B3JjlbR4ReSapO7ESuBRyJiaiWv1x7GjV/D+jVdWDC3V9qhZFZtXTMHjd7A7+8czudPPpotb9byyc8sSjusVG1+vZZ5j+zF156axTf//BzbNtcw897+O87PeqA/h//z6hQjbGdR4tZBKprIIqIpIg4H9gHGSTrkne+RdJakGZJmbGveUslwSjLmyDc4+vg1/OqxaXztRy9y6FHr+coPXkw7rExZ/bcGVq/syktz+gDw9KODOHD0hpSjSteCp/uw1/Ct9OzfSG19cMhJ63htZuGP4aa1dSx5vgejj1+fbpDtSBElbR2lQ+5aRsR6SU8AE4A57zg3CZgE0KduQOr11RuvGMGNV4wA4L3j1vMvn1nK5V8dnXJU2bJuTVdWrWhg2H6bWPpaDw4ft5bFC3ukHVaq+g7dyuLnerLtzRrqG5pZ8Exv9nnvJgBmP9SP0cevp75r6v+8208nums5ENieJLFuwIeAyyp1PetY1142mq9eMpu6umDF0m5ceXHnHqKy7xGbeO+H1/LTjxxCTV0wdMxmjjq5MLPm+Qf6M/7cZSlH2I4C6EQPHxkC3CSplkIT9o6IeLCC12t3s6f1Zfa0vmmHkUkLX+7FBafm4iZ0hznxwqWceOHSdx0/+/b5O3l3fomObTaWopJ3LV+gsCyHmVWb5mxVyTyy38zK08malmZWpTpN09LMqpgTmZnlWyeaNG5mVarlKUoZ4kRmZmVzH5mZ5Z8TmZnlWgDNTmRmlmvu7DezauBEZma5FkBTtob2O5GZWZkCwonMzPLOTUszyzXftTSzquAamZnlnhOZmeVaBDRl67F2TmRmVj7XyMws95zIzCzfwnctzSznAsIDYs0s9zxFycxyLcKPgzOzKuDOfjPLu3CNzMzyzQsrmlneedK4meVdAJGxKUo1aQdgZjkTycKKpWxFSJog6SVJCyRdtLshuUZmZmWLdmhaSqoFrgE+BCwBpku6PyLmlVuWa2RmVr72qZGNAxZExMKI2AbcDkzcnXAUGbr7IGkV8FracbQyAFiddhAZ5u+nuKx9R/tFxMA9KUDSwxR+r1I0AFta7U+KiElJOZ8AJkTEZ5P904CjIuL8cmPKVNNyT7/g9iZpRkSMTTuOrPL3U1w1fkcRMSHtGN7JTUszS8tSYHir/X2SY2VzIjOztEwHRkoaIakL8Cng/t0pKFNNywyalHYAGefvpzh/R7sQEY2SzgcmA7XALyNi7u6UlanOfjOz3eGmpZnlnhOZmeWeE9lOSPqlpJWS5qQdSxZJGi7pCUnzJM2VdEHaMWWJpAZJ0yQ9n3w/30k7pmrnPrKdkHQcsBG4OSIOSTuerJE0BBgSETMl9QKeBT66O1NLqpEkAT0iYqOkeuBp4IKI+HPKoVUt18h2IiKeAtamHUdWRcTyiJiZvN4AzAeGpRtVdkTBxmS3PtlcY6ggJzLbI5L2B44ApqYcSqZIqpU0C1gJPBIR/n4qyInMdpuknsDdwBcj4o2048mSiGiKiMMpjFYfJ8ldFBXkRGa7Jen7uRu4JSLuSTuerIqI9cATQObmJ1YTJzIrW9KZfQMwPyKuSDuerJE0UFLf5HU3CuttvZhqUFXOiWwnJN0G/AkYJWmJpDPTjiljjgVOA46XNCvZ/iHtoDJkCPCEpBcozCd8JCIeTDmmqubhF2aWe66RmVnuOZGZWe45kZlZ7jmRmVnuOZGZWe45keWIpKZkqMMcSXdK6r4HZd2YPMUGSddLGtPGe8dLev9uXOMvkt71tJ1dHX/Heza2dX4n7/+2pK+UG6NVByeyfHkzIg5PVuTYBpzT+qSk3Vq6PCI+W2TlivFA2YnMrKM4keXXH4CDktrSHyTdD8xLJiv/UNJ0SS9IOhsKo/ElXZ08nv5RYO+WgiQ9KWls8nqCpJnJWlqPJZPCzwEuTGqDf5eMXL87ucZ0Sccmn+0vaUqyBtf1gIr9EpL+R9KzyWfOese5K5Pjj0kamBw7UNLDyWf+IGl0u3yblmt++EgOJTWvDwMPJ4eOBA6JiEVJMng9It4nqSvwR0lTKKxQMQoYAwwC5gG/fEe5A4FfAMclZfWLiLWSrgU2RsTlyftuBa6MiKcl7Uvh4RH/C7gYeDoivivpH4FSZkR8JrlGN2C6pLsjYg3QA5gRERdK+r9J2edTeJjHORHxiqSjgJ8Bx+/G12hVxIksX7olS8NAoUZ2A4Um37SIWJQcPxE4tKX/C+gDjASOA26LiCZgmaTHd1L+0cBTLWVFxK7WZDsBGFOYcglA72QljOOAjyef/Z2kdSX8Tl+Q9LHk9fAk1jVAM/Db5PhvgHuSa7wfuLPVtbuWcA2rck5k+fJmsjTMDsl/6E2tDwGfj4jJ73hfe86FrAGOjogtO4mlZJLGU0iKx0TEZklPAg27eHsk113/zu/AzH1k1WcycG6yzA6S3iOpB/AU8G9JH9oQ4AM7+eyfgeMkjUg+2y85vgHo1ep9U4DPt+xIOjx5+RRwSnLsw8BeRWLtA6xLkthoCjXCFjVAS63yFApN1jeARZL+NbmGJB1W5BrWCTiRVZ/rKfR/zVTh4SnXUah53wu8kpy7mcLqHm8TEauAsyg0457nrabdA8DHWjr7gS8AY5ObCfN46+7pdygkwrkUmpiLi8T6MFAnaT5wKYVE2mIThQUJ51DoA/tucvxU4MwkvrnAxBK+E6tyXv3CzHLPNTIzyz0nMjPLPScyM8s9JzIzyz0nMjPLPScyM8s9JzIzy73/DxaZQNpdQ0a9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Results for highest accuracy model\n",
      "Trained ShallowConvNet on single session using train/test split and got accuracy of: 0.9548611111111112\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAb20lEQVR4nO3de5RU1Zn38e9T1c2thYYGRK6KihB0IjDE64TBS17JZalxZTJqYlwzOsYo0ZiYN5qY14yJvhqNxngNaoyZKHgfiEExIo7Cq1xFRC6CIoiA0NyRS3dXPe8f57S0BLqq6Ko+5xS/z1pndZ1zqvZ5utbqp/feZ+99zN0REUmyVNQBiIi0lBKZiCSeEpmIJJ4SmYgknhKZiCReRdQBNNWtJu2H9a2MOozYendeh6hDiD1L6X9zc3Zkt1HnO60lZZxxSpWv35DJ672z5+2a5O6jWnK9fMQqkR3Wt5IZk/pGHUZsndFrSNQhxF6qQ1XUIcTaG9ufa3EZtRsyTJ/UJ6/3VvZ8r1uLL5iHWCUyEUkCJ+PZqIP4DCUyESmIA1niNZBeiUxECpZFNTIRSTDHqVfTUkSSzIGMmpYiknTqIxORRHMgE7NVc5TIRKRg8eohUyITkQI5rj4yEUk2d6iPVx5TIhORQhkZWjRds+iUyESkIA5kVSMTkaRTjUxEEi0YEKtEJiIJ5kC9x2vdNyUyESmIY2Ritri0EpmIFCzralqKSIKpj0xEyoCRUR+ZiCRZsEKsEpmIJJi7UefpqMP4DCUyESlYVn1kIpJkQWe/mpYikmjq7BeRhFNnv4iUhYwGxIpIkjlGvccrdcQrGhGJvTh29scrGhGJPcfIeH5bLmZ2lZm9Y2bzzWysmbUzs/5mNt3MlprZ42bWJlc5SmQiUrAsqby25phZb+AKYLi7HwOkgXOBW4A73P1IYCNwUa54lMhCzz7YjUtOGch/jBzIMw90/8y5p+7vzhm9hrB5fbxGM0dp+MgtPPjaIh6etpBvjv446nBipbJNlt8+NY97Jszl/olv8u0rVkQdUlG5Q8ZTeW15qADam1kF0AFYDZwKPBWefwQ4O59CSsLM/gB8DVgbZtvY+mBRO55/tCu/++u7VLZxfnr+ERx/+mZ6969j7UeVzPmfjhzcuy7qMGMjlXIuv+kjrj33cGpXV3LXxCW8MamaFUvaRR1aLNTXGdd852h2bk+Trshy27j5zHq1C4vmdow6tKIIOvvz/qfezcxmNdkf4+5jANz9IzO7DVgB7ABeBGYDm9y9IXz/SqB3rouUskb2R2BUCcsvmhVL2jJo6HbadXDSFfD5E7cxbWJnAH7/i95cdN0qLF53myM1cOh2Vn3QhjUr2tJQn+KV8Z058YzNUYcVI8bO7cEfekWFU1HhxOzB3C2WIZXXBtS6+/Am25jGMsysC3AW0B/oBVSxnzmjZInM3V8FNpSq/GI6bNBO5s+oYsuGNDu3GzNf7sS6VZX8vxc60e2Qeo44emfUIcZK10PqWbdqd/9r7epKuvWsjzCi+EmlnLsnzGXsGzN5c1o1i98qj9oYBDWyrOe35XA6sMzd17l7PfAMcDLQOWxqAvQBPspVkIZfAP0G7OKbl63l2vOOoF2HLIcfvYP6OmPcXT34v2Pfizo8SaBs1hh95hCqOjbw83sXceiAT1i+pCrqsIqmSMMvVgAnmFkHgqblacAsYArwDWAccCEwPldBkXf2m9klZjbLzGatW5+JLI5R52/gnknv8ptnl3JQdYZDB+5kzYo2fO/0QXznuMGsW13J5WcMZMNa5f71ayrp3mt3n2G3nvXUrq6MMKL4+mRrBfOmVzN8xKaoQyma4LmWqby2Zstxn07QqT8HeJsgH40BfgL80MyWAl2Bh3LFFPlfZdhmHgMw/Nh2kfUkbKqtoHO3BtaurGTaxGrufG4JX7+49tPz3zluMHc9v5jqrtEl27hYPLcDvfvX0aPvLtavqWTkWZu4+fJDow4rNqpr6mmoNz7ZWkGbthmGnrSJJx/I2V+dIMV70ri7Xw9cv8fh94HjCikn8kQWFzdcfBhbN1aQrnRG37SSg6qVsPYlmzHu+VlvbnrsfVJpeHFcDcvf1R3LRl2613H1r5eSSjmWcl57vhszptREHVbRBI+Di9dQpFIOvxgLjCS4/boSuN7dc1YRo3L7fy9t9vyfZixopUiSYebLnZj5cqeow4ilDxZXMfqsY6MOo2TcLWezsbWVLJG5+3mlKltEoqX1yEQk0YL1yOI1sFKJTEQKpBViRSThguEXqpGJSIIVONeyVSiRiUjBtGa/iCRasIyPmpYiknDqIxORRAtWv1DTUkQSLJiipEQmIommGpmIlAGN7BeRRNNdSxEpC2paikiiNa7ZHydKZCJSEAcaVCMTkaRT01JEki2/R721KiUyESmIFlYUkbKgGpmIJJoWVhSRxHOMhqw6+0Uk4dRHJiLJ5mpaikjCqY9MRMqCEpmIJJpjZNTZLyJJp85+EUk0V2e/iJQDVyITkWTTpHERKQOqkTXj3XkdOKPXkKjDiK37l0+NOoTYu3zYmVGHEG87W56A3CGTVSITkYTTXUsRSTQnfk3LeI1qE5EECDr789lylmTW2cyeMrNFZrbQzE40sxoz+5uZLQl/dslVjhKZiBTMPb8tD3cCL7j7IOBYYCFwDTDZ3QcAk8P9ZimRiUjB3C2vrTlmVg2MAB4KyvQ6d98EnAU8Er7tEeDsXPGoj0xEChLctcy7DtTNzGY12R/j7mPC1/2BdcDDZnYsMBu4Eujh7qvD96wBeuS6iBKZiBQsz2YjQK27D9/HuQpgGPB9d59uZneyRzPS3d3Mcl5NTUsRKVgxmpbASmClu08P958iSGwfm1lPgPDn2lwFKZGJSEGc/JJYrkTm7muAD81sYHjoNGABMAG4MDx2ITA+V0xqWopIwfJvWeb0feBRM2sDvA/8G0EF6wkzuwhYDnwzVyFKZCJSGAcv0hQld58L7K0P7bRCylEiE5GCxW1kvxKZiBSsgLuWrWKficzM7qKZprC7X1GSiEQk1uI417K5GtmsZs6JyIHKgaQkMnd/pOm+mXVw9+2lD0lE4i5uTcuc48jC2egLgEXh/rFmdm/JIxORmDI8m9/WWvIZEPtb4AxgPYC7v0Uw0VNEDlSe59ZK8rpr6e4fmn0mu2ZKE46IxJ4nq7O/0YdmdhLgZlZJMDt9YWnDEpFYS1ofGXApcDnQG1gFDAn3ReSAZXlurSNnjczda4FvtUIsIpIU2agD+Kx87loebmZ/MbN1ZrbWzMab2eGtEZyIxFDjOLJ8tlaST9PyMeAJoCfQC3gSGFvKoEQk3oq4Zn9R5JPIOrj7f7l7Q7j9GWhX6sBEJMaSMvzCzGrCl8+b2TXAOILQ/hWY2AqxiUhcJWj4xWyCxNUY8XebnHPg2lIFJSLxlnsV/dbV3FzL/q0ZiIgkhBu04vSjfOQ1st/MjgEG06RvzN3/VKqgRCTmklIja2Rm1wMjCRLZRODLwFRAiUzkQBWzRJbPXctvEKyfvcbd/43gsebVJY1KROItKXctm9jh7lkzazCzTgTPmOtb4rgiNXzkFi795SrSKef5sTU8cXfOBx0fEF56sBfTxvXADHoN2s6Ft77Le7M78fSN/cnUG/3+YRsX/HoJaS2gzsMvvM6O7WkyGSObMa48d1/PqE2gJC2s2MQsM+sMPEBwJ3Mb8HquD5lZX4LmZw+CX32Mu9+5/6G2jlTKufymj7j23MOpXV3JXROX8MakalYsObCHzm1c04YpD/fi+slzaNMuy5jLBjJj/ME8d0c/fvDY2/Q4fCcTftOPN57qwcnnfhx1uLFwzb8PYcumNlGHURJxu2uZs2np7pe5+yZ3vx/4EnBh2MTMpQH4kbsPBk4ALjezwS0Lt/QGDt3Oqg/asGZFWxrqU7wyvjMnnrE56rBiIZsx6nemyDRA/Y40bTtkSFdm6XH4TgA+98VNzHm+a8RRSquIWdNyn4nMzIbtuQE1QEX4ulnuvtrd54SvtxIs/dO7WIGXStdD6lm3avd/0drVlXTrWR9hRPHQ5ZA6Tr/kI3564hf4yReOp13HBv7xa7VkM8byeQcBMGdiNzaubhtxpPHgDr/6/Vvc+fhMRn1jVdThFJ15fltraa5p+Ztmzjlwar4XMbPDgKHA9L2cuwS4BKAdHfItUlrZJ5vTzHuxhl9NnUmHThnGXDaIGc925+K7FvPkDf2pr0sx+IsbSaVj1uaIyI8vHMb6tW2prqnjxjFzWbmsA/Nnd446rOJJSh+Zu59SjAuY2UHA08AP3H3LXq4zBhgD0MlqIv8rWL+mku696j7d79azntrVlRFGFA+Lpnama9+ddOzaAMDQUet5b3Ynjj9nHVc/9TYAC17tzMfL2kcZZmysXxvUTDdvaMPrk7tz1DFbyieRtXKzMR/5DL/Yb+GKsk8Dj7r7M6W8VrEsntuB3v3r6NF3FxWVWUaetYk3XtRok5peu1j2ZkfqdqRwh0XTqul55Ha21AZJvn6XMem+Poz41pqII41e2/YZ2ndo+PT10JM2sHxpVcRRFVnM+shKdqPcgkX+HwIWuvvtpbpOsWUzxj0/681Nj71PKg0vjqth+bsH9h1LgP5DtzHsK+u58atDSKedvkd/wj+dv4YJtx3K25NrcIcR317DoJN1Y6RL1zqu+21QS02nnVcm9mD2tPK6CWIxW1jRvESLBpnZPwGvAW+zez3Jn7r7PlfO6GQ1frydVpJ4ysH9y6dGHULsXT7szKhDiLXXNz3D5vp1Lergatu3r/e58qq83vv+j380291LPogunylKRrDU9eHufoOZ9QMOcfcZzX3O3afSmot2i0iraO07kvnIp4/sXuBE4LxwfytwT8kiEpH4i9lS1/n0kR3v7sPM7E0Ad99oZuU5XFlE8hOzGlk+iazezNKEoZtZd2L3DBURaU1xa1rmk8h+BzwLHGxmNxKshnFdSaMSkfjy+N21zOe5lo+a2WyCpXwMONvd9aRxkQNZ0mpk4V3K7cBfmh5z9xWlDExEYixpiQz4K7sfQtIO6A8sBo4uYVwiEmNx6yPLZxmff3D3z4c/BwDHkcd6ZCIi+TCztJm9aWbPhfv9zWy6mS01s8fzGSVR8FzLcGme4/cjXhEpF8Wda3klwTJfjW4B7nD3I4GNwEW5Csinj+yHTXZTwDCg/BZYEpH8FPGupZn1Ab4K3Aj8MJxJdCpwfviWR4BfAPc1V04+fWQdm7xuIOgze7rAeEWknORf2+pmZrOa7I8Jl+5q9Fvgf7M7z3QFNrl7Q7i/kjwWZG02kYUDYTu6+9X5Ri0i5c0oqLO/dl+Txs3sa8Bad59tZiNbEtM+E5mZVbh7g5md3JILiEgZKs5dy5OBM83sKwQjIjoBdwKdG/MP0Af4KFdBzXX2N65uMdfMJpjZBWZ2TuPWwl9ARJIqz/X6c9Xa3P1ad+/j7ocB5wIvu/u3gCkEM4gALgTG5wopnz6ydsB6gg64xvFkDiRixVcRKYHSTlH6CTDOzH4FvEmwQGuzmktkB4d3LOezO4E1itlwOBFpTcUeEOvurwCvhK/fJxivmrfmElkaOIi9L46oRCZyIItZBmguka129xtaLRIRSYYYPkWpuUSmZapFZK/iNteyuUSmp4CIyN4lJZG5+4bWDEREkiNxCyuKiHxGwvrIRET+jhG/DnQlMhEpnGpkIpJ0SbprKSKyd0pkIpJoSXwcnIjI31GNTESSTn1kIpJ8SmT7ZqkUqQ5VUYcRW987SrPGcnnqvYlRhxBr//zlzUUpRzUyEUk2p9QLKxZMiUxEClLgw0dahRKZiBROiUxEks48XplMiUxECqPVL0SkHKiPTEQST1OURCT5VCMTkUTL4ynirU2JTEQKp0QmIkmmAbEiUhYsG69MpkQmIoXRODIRKQcafiEiyacamYgknTr7RSTZHNCkcRFJOvWRiUiiaRyZiCSfu5qWIpJ8qpGJSPLFLJGlog5ARJLHPL+t2TLM+prZFDNbYGbvmNmV4fEaM/ubmS0Jf3bJFY8SmYgUxoGM57c1rwH4kbsPBk4ALjezwcA1wGR3HwBMDvebpUQmIgUrRo3M3Ve7+5zw9VZgIdAbOAt4JHzbI8DZueJRH5mIFK7Idy3N7DBgKDAd6OHuq8NTa4AeuT6vRCYiBSvgrmU3M5vVZH+Mu4/5TFlmBwFPAz9w9y1m9uk5d3ez3FdTIhORwhS2jE+tuw/f10kzqyRIYo+6+zPh4Y/NrKe7rzaznsDaXBdRH5mIFMQAy3heW7PlBFWvh4CF7n57k1MTgAvD1xcC43PFpBqZiBSsSE8aPxm4AHjbzOaGx34K3Aw8YWYXAcuBb+YqSIlMRApTpBVi3X0qQQVvb04rpCwlsj1Utsly62PzqWyTJV3hTH2hK3/+Xb+ow4qVbj138ePfvE/nbvXgxsSx3Rn/x0OiDityzz14CC+NPRh3+NL5a/naxWt45Jf9mPVSFyoqsxxy6C5G3/4eVdWZqENtoQNorqWZtQNeBdqG13nK3a8v1fWKpb7OuOY7R7Nze5p0RZbbxs1n1qtdWDS3Y9ShxUa2wXjgxn4sfaeK9lUZ7vrLfN6cWs2Kpe2jDi0yKxa156WxB3PLc/OpqMzyy29/jn88bSPHjtjMt69dQboC/uvGfjxzd28u+NmKqMNtsbjNtSxlZ/8u4FR3PxYYAowysxNKeL0iMXZuTwNQUeFUVHjc/vlEbsO6Nix9pwqAHZ+k+XBpe7oeUhdxVNFaubQ9A4Zso237LOkKOPqELUx/voYh/7yZdFhdOGrYVtavbhNtoMXSuAJGrq2VlCyReWBbuFsZbolICamUc/eEuYx9YyZvTqtm8Vuqje1Lj967OGLwdhbPPSjqUCLVb+B2Fs7oyNaNFezakWLOy52pXdX2M++Z/PjBDD1lUzQBFpMX565lMZW0j8zM0sBs4EjgHnefXsrrFUs2a4w+cwhVHRv4+b2LOHTAJyxfUhV1WLHTrkOG6+5bwu9/2Y/t29JRhxOpPgN2cvZlq7jh/EG07ZDlsKO3k0rv/kN+6ne9SKedEefURhhlEcWsSlLSRObuGWCImXUGnjWzY9x9ftP3mNklwCUA7SxeyeKTrRXMm17N8BGblMj2kK7I8vP7ljBlfFemTaqJOpxYOP28dZx+3joAHr25L117Bs3tl5/ozuyXuvCLxxdi+7pHlzBFGn5RNK0yINbdNwFTgFF7OTfG3Ye7+/A21q41wmlWdU09VR0bAGjTNsPQkzbx4fsHbif23jlX3bKMFUvb88xDPaMOJjY21wb1gnUfteGN52v44tm1vDmlmvH39eSahxfTtn3MFrpviZj1kZXyrmV3oN7dN5lZe+BLwC2lul6xdOlex9W/Xkoq5VjKee35bsyYohpHU0cP38bp56xn2aL23PPXoIL9x1v7MPOVztEGFrFbLzmKrRsrSFc4/3HjMqqqMzx4XX/q64wbzvscAEcN28Z3b14WcaQt5EDMcnIpm5Y9gUfCfrIU8IS7P1fC6xXFB4urGH3WsVGHEWvvzOrIqP7HRR1G7PzqmQV/d+yeaXNbP5ASMzx2TcuSJTJ3n0ewLIeIlJtsvKpkGtkvIoU5wJqWIlKmDpimpYiUMSUyEUm2A2jSuIiUqcanKMWIEpmIFEx9ZCKSfEpkIpJoDmSVyEQk0dTZLyLlQIlMRBLNgUy8hvYrkYlIgRxciUxEkk5NSxFJNN21FJGyoBqZiCSeEpmIJJo7ZOL1tHQlMhEpnGpkIpJ4SmQikmyuu5YiknAOrgGxIpJ4mqIkIonmrsfBiUgZUGe/iCSdq0YmIsmmhRVFJOk0aVxEks4Bj9kUpVTUAYhIwni4sGI+Ww5mNsrMFpvZUjO7Zn9DUo1MRArmRWhamlkauAf4ErASmGlmE9x9QaFlqUYmIoUrTo3sOGCpu7/v7nXAOOCs/QnHPEZ3H8xsHbA86jia6AbURh1EjOn7yS1u39Gh7t69JQWY2QsEv1c+2gE7m+yPcfcxYTnfAEa5+8Xh/gXA8e4+utCYYtW0bOkXXGxmNsvdh0cdR1zp+8mtHL8jdx8VdQx7UtNSRKLyEdC3yX6f8FjBlMhEJCozgQFm1t/M2gDnAhP2p6BYNS1jaEzUAcScvp/c9B3tg7s3mNloYBKQBv7g7u/sT1mx6uwXEdkfalqKSOIpkYlI4imR7YWZ/cHM1prZ/KhjiSMz62tmU8xsgZm9Y2ZXRh1TnJhZOzObYWZvhd/Pf0YdU7lTH9lemNkIYBvwJ3c/Jup44sbMegI93X2OmXUEZgNn78/UknJkZgZUufs2M6sEpgJXuvsbEYdWtlQj2wt3fxXYEHUcceXuq919Tvh6K7AQ6B1tVPHhgW3hbmW4qcZQQkpk0iJmdhgwFJgecSixYmZpM5sLrAX+5u76fkpIiUz2m5kdBDwN/MDdt0QdT5y4e8bdhxCMVj/OzNRFUUJKZLJfwr6fp4FH3f2ZqOOJK3ffBEwBYjc/sZwokUnBws7sh4CF7n571PHEjZl1N7PO4ev2BOttLYo0qDKnRLYXZjYWeB0YaGYrzeyiqGOKmZOBC4BTzWxuuH0l6qBipCcwxczmEcwn/Ju7PxdxTGVNwy9EJPFUIxORxFMiE5HEUyITkcRTIhORxFMiE5HEUyJLEDPLhEMd5pvZk2bWoQVl/TF8ig1m9qCZDW7mvSPN7KT9uMYHZvZ3T9vZ1/E93rOtufN7ef8vzOzqQmOU8qBEliw73H1IuCJHHXBp05Nmtl9Ll7v7xTlWrhgJFJzIRFqLEllyvQYcGdaWXjOzCcCCcLLyrWY208zmmdl3IRiNb2Z3h4+nfwk4uLEgM3vFzIaHr0eZ2ZxwLa3J4aTwS4GrwtrgF8OR60+H15hpZieHn+1qZi+Ga3A9CFiuX8LM/tvMZoefuWSPc3eExyebWffw2BFm9kL4mdfMbFBRvk1JND18JIHCmteXgRfCQ8OAY9x9WZgMNrv7F8ysLTDNzF4kWKFiIDAY6AEsAP6wR7ndgQeAEWFZNe6+wczuB7a5+23h+x4D7nD3qWbWj+DhEZ8DrgemuvsNZvZVIJ8ZEf8eXqM9MNPMnnb39UAVMMvdrzKz/xOWPZrgYR6XuvsSMzseuBc4dT++RikjSmTJ0j5cGgaCGtlDBE2+Ge6+LDz+v4DPN/Z/AdXAAGAEMNbdM8AqM3t5L+WfALzaWJa772tNttOBwcGUSwA6hSthjADOCT/7VzPbmMfvdIWZfT183TeMdT2QBR4Pj/8ZeCa8xknAk02u3TaPa0iZUyJLlh3h0jCfCv+gP2l6CPi+u0/a433FnAuZAk5w9517iSVvZjaSICme6O7bzewVoN0+3u7hdTft+R2IqI+s/EwCvhcus4OZHWVmVcCrwL+GfWg9gVP28tk3gBFm1j/8bE14fCvQscn7XgS+37hjZkPCl68C54fHvgx0yRFrNbAxTGKDCGqEjVJAY63yfIIm6xZgmZn9S3gNM7Njc1xDDgBKZOXnQYL+rzkWPDzl9wQ172eBJeG5PxGs7vEZ7r4OuISgGfcWu5t2fwG+3tjZD1wBDA9vJixg993T/yRIhO8QNDFX5Ij1BaDCzBYCNxMk0kafECxIOJ+gD+yG8Pi3gIvC+N4BzsrjO5Eyp9UvRCTxVCMTkcRTIhORxFMiE5HEUyITkcRTIhORxFMiE5HEUyITkcT7/7OguHtddRWkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################################\n",
    "# SINGLE TRIAL SHALLOWCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Configure\n",
    "retrain_model = False\n",
    "tensorboard_name = \"ShallowCN_singlesession_C0_100hz_08drop\"\n",
    "best_model_filename = f\"./saved_variables/6/ShallowConvNet/{tensorboard_name}\"\n",
    "\n",
    "# Get train/test split of data from one session\n",
    "X_train, X_test, y_train, y_test = train_test_split(mne_fixed_window_epochs_data, ohe_labels, \n",
    "                                                    test_size = 0.3,\n",
    "                                                    shuffle= True,\n",
    "                                                    stratify= ohe_labels,                                                    \n",
    "                                                    random_state=98)\n",
    "\n",
    "\n",
    "# Train on train/test split of data from one session\n",
    "## Note: the model is forced to use GPU, if GPU is not available replace with what is available e.g. /cpu:0\n",
    "if (retrain_model): # Retrain or not\n",
    "    with tf.device('/gpu:0'):\n",
    "        keras_shallowcn_model.fit(\n",
    "            x= X_train,\n",
    "            y= y_train,\n",
    "            batch_size= 128, # Default: 32\n",
    "            epochs= 500, # Default: 500 (EEGNet paper)\n",
    "            verbose= 1, # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "            #callbacks= [tensorboard_callback(\"EEGNet_raw_signal_0.5s_100samps_50kernlen_02nr\")], # To be used for TF Board\n",
    "            callbacks= [tensorboard_callback(tensorboard_name),\n",
    "                        lowest_loss_model_save_callback(best_model_filename),\n",
    "                        highest_accuracy_model_save_callback(best_model_filename)],\n",
    "            validation_split= 0.3,\n",
    "            shuffle= True,\n",
    "            sample_weight= None, # Can be interesting due to time series\n",
    "            use_multiprocessing=True, # Done for faster speed\n",
    "            workers= 4 # Done for faster speed\n",
    "            )\n",
    "\n",
    "# Convert labels back to original\n",
    "y_test = ohe.inverse_transform(y_test)\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"Results for lowest loss model\")\n",
    "keras_shallowcn_model = load_lowest_loss_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_shallowcn_model.predict(X_test)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained ShallowConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"\\n\\nResults for highest accuracy model\")\n",
    "keras_shallowcn_model = load_highest_accuracy_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_shallowcn_model.predict(X_test)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained ShallowConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Remove unused variables\n",
    "del X_train\n",
    "del X_test\n",
    "del y_train\n",
    "del y_test\n",
    "del accuracy\n",
    "del y_pred\n",
    "del retrain_model\n",
    "del tensorboard_name\n",
    "del best_model_filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d4a6c53d",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAUP\n",
    "####################################################\n",
    "\n",
    "# delete unused variables\n",
    "del keras_shallowcn_model\n",
    "del mne_fixed_window_epochs_data\n",
    "del ohe\n",
    "del ohe_labels\n",
    "del mne_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d4cd19",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Fixed window classification: Multi trial training with unseen trial testing | ShallowConvNet | three class MI task | 100Hz input signal\n",
    "\n",
    "Having equally good if not better performance then EEGNet, it is studied if the inter-session performance is also as good or better.\n",
    "The scores given are best loss/ best accuracy.\n",
    "\n",
    "**Remember the results for subject `C`**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|-----------------------------------|\n",
    "| 0              | 1 + 2           | 0.782                        | 0.916 / 0.920             | 0.903 / 0.910                     |\n",
    "| 1              | 0 + 2           | 0.635                        | 0.852 / 0.860             | 0.782 / 0.787                     |\n",
    "| 2              | 1 + 2           | 0.347                        | 0.607 / 0.594             | 0.509 / 0.489                     |\n",
    "| 0              | 1               | 0.667                        | 0.891 / 0.860             | 0.872 / 0.872                     |\n",
    "| 1              | 0               | 0.662                        | 0.794 / 0.783             | 0.713 / 0.731                     |\n",
    "\n",
    "**For subject `B` these results are**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|-----------------------------------|\n",
    "| 0              | 1 + 2           | 0.409                        | 0.498 / 0.489             | 0.447 / 0.448                     |\n",
    "| 1              | 0 + 2           | 0.497                        | 0.640 / 0.654             | 0.570 / 0.571                     |\n",
    "| 2              | 1 + 2           | 0.502                        | 0.677 / 0.660             | 0.602 / 0.601                     |\n",
    "\n",
    "**For subject `E` the results are**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|-----------------------------------|\n",
    "| 0              | 1 + 2           | 0.398                        | 0.714 / 0.713             | 0.656 / 0.681                     |\n",
    "| 1              | 0 + 2           | 0.371                        | 0.667 / 0.646             | 0.641 / 0.713                     |\n",
    "| 2              | 1 + 2           | 0.475                        | 0.734 / 0.688             | 0.699 / 0.712                     |\n",
    "\n",
    "As expected, the inter-session performance is worse then EEGNet due to the feauture engineering being fixed in a way.\n",
    "In the next section we study the Deep Convolution Network variant of this architecture to see if this improves things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6f9f3465",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 0 ... 666799  =      0.000 ...  3333.995 secs...\n",
      "Reading 0 ... 681199  =      0.000 ...  3405.995 secs...\n",
      "Reading 0 ... 669399  =      0.000 ...  3346.995 secs...\n",
      "Loaded test epochs and extracted labels\n",
      "Loaded train epochs and extracted labels\n",
      "Loaded train epochs and extracted labels\n",
      "\n",
      "\n",
      "Concatenated the train epochs\n",
      "Total amount of train labels: 1919\n",
      "\n",
      "\n",
      "Got the test epochs\n",
      "Total amount of train labels: 960\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Lennert\\AppData\\Local\\Temp\\ipykernel_16324\\2395505687.py:29: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.\n",
      "  mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
      "C:\\Users\\Lennert\\AppData\\Local\\Temp\\ipykernel_16324\\2395505687.py:34: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.\n",
      "  mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OHE test and train labels, train labels:\n",
      "[[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]]\n",
      "Train labels OHE match regular labels: True\n",
      "Test labels OHE match regular labels: True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# PREPPING THE DATA\n",
    "####################################################\n",
    "\n",
    "# Get data and choose test trial\n",
    "mne_raws = load_mne_raws(subject= \"C\")\n",
    "test_trial = mne_raws[0]\n",
    "\n",
    "# Init variables\n",
    "epochs = []\n",
    "labels_train = []\n",
    "\n",
    "for single_mne_raw in mne_raws:\n",
    "    # Load epochs from raw\n",
    "    mne_fixed_window_epochs = CLA_dataset.get_usefull_epochs_from_raw(single_mne_raw, start_offset=-1.5, end_offset=1.5)['task/neutral', 'task/left', 'task/right']\n",
    "    with io.capture_output():\n",
    "        mne_fixed_window_epochs.load_data()\n",
    "    \n",
    "    if (single_mne_raw == test_trial):\n",
    "        mne_fixed_window_epochs_test = mne_fixed_window_epochs\n",
    "        labels_test = mne_fixed_window_epochs.events[:, -1]\n",
    "        print(\"Loaded test epochs and extracted labels\")\n",
    "    else:  \n",
    "        epochs.append(mne_fixed_window_epochs)\n",
    "        labels_train.extend(mne_fixed_window_epochs.events[:, -1])\n",
    "        print(\"Loaded train epochs and extracted labels\")\n",
    "    \n",
    "# Make single epoch object for training\n",
    "mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
    "print(\"\\n\\nConcatenated the train epochs\")\n",
    "print(f\"Total amount of train labels: {len(labels_train)}\")\n",
    "    \n",
    "# Show test epoch object\n",
    "mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
    "print(\"\\n\\nGot the test epochs\")\n",
    "print(f\"Total amount of train labels: {len(labels_test)}\")\n",
    "\n",
    "# Go to 2D representation\n",
    "labels_train = np.array(labels_train).reshape(-1, 1)\n",
    "labels_test = np.array(labels_test).reshape(-1, 1)\n",
    "\n",
    "# One Hot Encode the labels\n",
    "ohe = OneHotEncoder()\n",
    "ohe_labels_train = ohe.fit_transform(labels_train).toarray()\n",
    "ohe_labels_test = ohe.transform(labels_test).toarray()\n",
    "\n",
    "\n",
    "# Show ohe labels\n",
    "print(f\"OHE test and train labels, train labels:\\n{ohe_labels_train[:10]}\")\n",
    "\n",
    "# Validate OHE\n",
    "print(f\"Train labels OHE match regular labels: {np.array_equal(ohe.inverse_transform(ohe_labels_train), labels_train)}\")\n",
    "print(f\"Test labels OHE match regular labels: {np.array_equal(ohe.inverse_transform(ohe_labels_test), labels_test)}\")\n",
    "\n",
    "# Get training data\n",
    "mne_fixed_window_epochs_train_data = mne_fixed_window_epochs_train.get_data(tmin=0.2, tmax=0.7)\n",
    "mne_fixed_window_epochs_test_data = mne_fixed_window_epochs_test.get_data(tmin=0.2, tmax=0.7)\n",
    "\n",
    "# Fix scaling sensitivity as MNE stores as data * 10e-6\n",
    "mne_fixed_window_epochs_train_data = mne_fixed_window_epochs_train_data * 1000000\n",
    "mne_fixed_window_epochs_test_data = mne_fixed_window_epochs_test_data * 1000000\n",
    "\n",
    "# Remove unused variables\n",
    "del epochs\n",
    "del mne_fixed_window_epochs\n",
    "del test_trial\n",
    "del single_mne_raw\n",
    "del mne_raws\n",
    "del mne_fixed_window_epochs_train\n",
    "del mne_fixed_window_epochs_test\n",
    "del labels_train\n",
    "del labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ee34bb11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_4 (InputLayer)        [(None, 21, 100, 1)]      0         \n",
      "                                                                 \n",
      " conv2d_4 (Conv2D)           (None, 21, 88, 40)        560       \n",
      "                                                                 \n",
      " conv2d_5 (Conv2D)           (None, 1, 88, 40)         33600     \n",
      "                                                                 \n",
      " batch_normalization_7 (Batc  (None, 1, 88, 40)        160       \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_7 (Activation)   (None, 1, 88, 40)         0         \n",
      "                                                                 \n",
      " average_pooling2d_5 (Averag  (None, 1, 8, 40)         0         \n",
      " ePooling2D)                                                     \n",
      "                                                                 \n",
      " activation_8 (Activation)   (None, 1, 8, 40)          0         \n",
      "                                                                 \n",
      " dropout_1 (Dropout)         (None, 1, 8, 40)          0         \n",
      "                                                                 \n",
      " flatten_1 (Flatten)         (None, 320)               0         \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 3)                 963       \n",
      "                                                                 \n",
      " activation_9 (Activation)   (None, 3)                 0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 35,283\n",
      "Trainable params: 35,203\n",
      "Non-trainable params: 80\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE SHALLOWCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Create the TensorFlow Keras model\n",
    "keras_shallowcn_model = ShallowConvNet(\n",
    "    nb_classes = 3, # int, number of classes to classify. \n",
    "    Chans = 21, # number of channels in the EEG data. \n",
    "    Samples = 100, # number of time points in the EEG data. (default: 128, paper: 250)\n",
    "    dropoutRate = 0.8, # dropout fraction. (default: 0.5)\n",
    "    conv_filters = 13, # Conv2D kernel size (default: 13, paper: 25)\n",
    "    strides = 7, # Stride size for average pooling layer (default: 7, paper: 15)\n",
    "    pool_size = 35 # Pool size for average pooling layer (default: 35, paper: 75)\n",
    "    )\n",
    "\n",
    "# Compile the model so it can be fitted\n",
    "# Loss and optimizer from EEGNet paper\n",
    "keras_shallowcn_model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics=[\"accuracy\"])\n",
    "\n",
    "# Show summary of the model\n",
    "keras_shallowcn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "70bf8dd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for lowest loss model\n",
      "Trained ShallowConvNet on single session using train/test split and got accuracy of: 0.903125\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAg40lEQVR4nO3deZgdZZn+8e/dnc6+kQRCVgIkLGELEFaBX0CBqOMAMwwKo6DiBEZERECFGQUZ/Y0LiCCKEwEFB3BQQBAQRIQJiywhhEASlkgI2ciedNZO+pxn/qhqaLHTfU7SnTqncn+uq66u81adquecq/vpd6l6SxGBmVke1WQdgJlZR3GCM7PccoIzs9xygjOz3HKCM7Pc6pR1AM0N6FcbI4bVZR1GxXp9WvesQ7Aqt4G1bIwGbc0xTjy2RyxbXihp3xemNTwcEeO35nxbo6IS3IhhdTz38LCsw6hYJw4ek3UIlU9b9bebe88W/7jVx1i6vMCzDw8tad+6QX8ZsNUn3AoVleDMrBoEhShmHURJnODMrCwBFKmOGwSc4MysbEVcgzOzHAqCTVXSRPVlImZWlgAKRElLayR1lfScpJckTZf0zbR8V0nPSpol6X8kdU7Lu6SvZ6XbR7QVqxOcmZWtSJS0tKEBOC4iDgDGAOMlHQ58F7gmIkYCK4Cz0/3PBlak5dek+7XKCc7MyhJAIaKkpdXjJNakL+vSJYDjgN+k5bcAJ6frJ6WvSbd/UGr9uiAnODMrW7HEBRggaXKzZULz40iqlTQVWAw8AvwFWBkRjeku84Ah6foQYC5Aun0V0L+1OD3IYGZliRL615pZGhFjN3usiAIwRlJf4B5gr62P8D1OcGZWlgjY1M6XwUXESkmPAUcAfSV1SmtpQ4H56W7zgWHAPEmdgD7AstaO6yaqmZVJFEpcWj2KtGNac0NSN+B4YCbwGHBquttZwL3p+n3pa9Ltf4o2piR3Dc7MyhJAsX1qcIOAWyTVklS27oyI+yXNAH4l6VvAi8BN6f43Ab+UNAtYDnyirRM4wZlZ2dqqnZUiIqYBB7ZQ/iZwaAvlG4B/KuccTnBmVpbkQt/qmLXFCc7MyhLApqiO7nsnODMrSyAKVTI+6QRnZmUrhpuoZpZD7oMzsxwTBffBmVkeJTP6OsGZWQ5FiI1Rm3UYJXGCM7OyFd0HZ2Z5lAwyuIlqZrnkQQYzyykPMphZrhV8oa+Z5VEgNkV1pI7qiNLMKoYHGcwstwK5iWpm+eVBhgq3cYO46B9GsmljDYVGOPqjqzjzkne49+YB3HPjjix8qwt3vvwyffoXAPjT3Ttw5493IgK69Shy/nfmsvs+GzL+FNmqqQl+9NDrLFtYxzfO2i3rcCpKj96NXHjVXEbsuYEI+MFFw5n5Qo+sw2oXEfgyEUk3A38HLI6IfTvqPFuqrkvwvV//hW49ijRugi+fPIpDjqtnn0PWctjx9XzlH0f+1f4DhzXw/btm0atvgef/1ItrvzKM6x54I6PoK8PJn1vK3De60r1nIetQKs6/XjmfyY/15lsTdqVTXZEu3YpZh9RukkGG6rhVqyPT8C+A8R14/K0iJTUxgMZNorBJSDByv/XsPGzj3+y/zyHr6NU3+UPe66B1LF1Yt03jrTQDBm3k0A/W8/vb+2UdSsXp3qvAfoet5aE7ku+mcVMNa+vz1VgqUFPSkrUO+9YjYpKkER11/PZQKMAXTtyTBW915mOfXspeB60r6X0P3dGPQ45d3cHRVbZzv7mAG781iO4981MzaS87D29g1bJOXHTN2+w2egNvTOvGDd8YQsP66qj1tCVQ1Ux4mX2KzVBtLdzwx9e47YUZvDa1O2+92rXN90x9qicP39Gfs/9twTaIsDId9qF6Vi7txKyXu2cdSkWqrYWR+63j/lsHcN6Je7JhXQ0f/8LirMNqV9VSg8s8AkkTJE2WNHnJsmz6cnr2KXDAkWt4/rFere735oyu/PDiYVzx89n07rf99juNPmQth59Qzy3PzuDSG+ZwwFFr+MqP5mQdVsVYurCOJQvreO3FZFDhyQf6MnK/9RlH1X6S56LWlLRkLfMIImJiRIyNiLE79t92VfiVy2pZsyo5X8N6MWVSL4aNbNjs/ovn1XHl53blkuvmMHT3ze+3Pfj5fw7ik2NHc9Zho/nPf92Fl57syffO3yXrsCrGiiV1LF3QmaG7J6PsY45azduvd8k4qvbUPk+23xby1fNZhuWL6rjqguEUi6JYhGM+tpLDj6/ntzcO4Nc37MTyxXWc+6G9OPS4ei68ei63XbMzq1fUcv2lwwCo7RRc/9DrGX8Kq1Q//voQvvqjOXSqC955uzNXf3l41iG1m+SxgdXRn6iI6JgDS3cA44ABwCLg8oi4qbX3jD2gazz38LAOiScPThw8JusQKp+yrzVUsmeLf6Q+lm/VlzRkn77x+TuPKmnff9/3gRciYuzWnG9rdFgTNSJOj4hBEVEXEUPbSm5mVj0KUVPS0hpJwyQ9JmmGpOmSLkjLr5A0X9LUdPlIs/dcKmmWpNckndhWnNttE9XMtkwyH1y71JQbgYsiYoqkXsALkh5Jt10TEVc131nSaOATwD7AYOCPkvaIiM2O+DnBmVmZ2mdG34hYCCxM11dLmgkMaeUtJwG/iogGYLakWcChwJ8394bMR1HNrLokl4mopAUY0HQZWLpMaOmY6U0BBwLPpkVfkDRN0s2SdkjLhgBzm71tHq0nRNfgzKw8Zd6LurStQQZJPYG7gC9FRL2kG4D/IMml/wFcDXx2S2J1gjOzsrXXdEmS6kiS220RcTdARCxqtv1nwP3py/lA88sshqZlm+UmqpmVJZkuSSUtrZEk4CZgZkT8oFn5oGa7nQK8kq7fB3xCUhdJuwKjgOdaO4drcGZWtna62f4DwKeAlyVNTcsuA06XNIakifoWcA5AREyXdCcwg2QE9rzWRlDBCc7MypTMJtIuo6hPQovXmzzYynu+DXy71HM4wZlZWZJbtaqjd8sJzszK1D41uG3BCc7MytZOdzJ0OCc4MytL0yhqNXCCM7OyuYlqZrlUTc9kcIIzs7IE0OganJnllZuoZpZP4SaqmeVUO0542eGc4MysbK7BmVkuNU14WQ2c4MysLIFoLHqQwcxyyn1wZpZP4SaqmeWU++DMLNec4MwslwJR8CCDmeWVBxnMLJfCgwxmlmfhBGdm+eSb7c0sx1yD2wKvT+vOiUMPzjqMivWLt/836xAq3md2G5d1CJUt2uEQAYWiE5yZ5ZRHUc0sl4LqaaJWx9V6ZlZBkkGGUpZWjyINk/SYpBmSpku6IC3vJ+kRSW+kP3dIyyXpOkmzJE2TdFBbkTrBmVnZIkpb2tAIXBQRo4HDgfMkjQa+BjwaEaOAR9PXAB8GRqXLBOCGtk7gBGdmZYtQSUvrx4iFETElXV8NzASGACcBt6S73QKcnK6fBNwaiWeAvpIGtXYO98GZWVmSUdSS60YDJE1u9npiREx8/06SRgAHAs8CAyNiYbrpHWBguj4EmNvsbfPSsoVshhOcmZWthOZnk6URMba1HST1BO4CvhQR9dJ7Nb+ICElbfHGLm6hmVrb2aKICSKojSW63RcTdafGipqZn+nNxWj4fGNbs7UPTss1ygjOzsgSlJbe2EpySqtpNwMyI+EGzTfcBZ6XrZwH3Nis/Mx1NPRxY1awp2yI3Uc2sbO1wQwTAB4BPAS9LmpqWXQZ8B7hT0tnAHOC0dNuDwEeAWcA64DNtncAJzszKExDtcKtWRDwJm70l4oMt7B/AeeWcwwnOzMpWLXcyOMGZWdnKGEXN1GYTnKQf0UpTOyK+2CERmVlFq6Z7UVurwU1uZZuZba8CqPYEFxG3NH8tqXtErOv4kMys0lVLE7XN6+AkHSFpBvBq+voAST/p8MjMrEKJKJa2ZK2UC31/CJwILAOIiJeAYzowJjOrdFHikrGSRlEjYm7z+8OAQseEY2YVL/IxyNBkrqQjgUjvG7uAZFoTM9teVUDtrBSlNFHPJbl6eAiwABhDmVcTm1neqMQlW23W4CJiKfDP2yAWM6sWxawDKE0po6i7SfqdpCWSFku6V9Ju2yI4M6tATdfBlbJkrJQm6u3AncAgYDDwa+COjgzKzCpbOz2TocOVkuC6R8QvI6IxXf4b6NrRgZlZBav2y0Qk9UtXfy/pa8CvSEL+OMm8TGa2vaqA5mcpWhtkeIEkoTV9knOabQvg0o4Kyswq25Y/JWHbau1e1F23ZSBmViVCUAG3YZWipDsZJO0LjKZZ31tE3NpRQZlZhav2GlwTSZcD40gS3IMkT5d+EnCCM9teVUmCK2UU9VSS+dHfiYjPAAcAfTo0KjOrbNU+itrM+ogoSmqU1JvkGYXD2npTtRq62wYuu2H2u693Ht7AL68azD037ZRhVNvesgWd+dmFe1C/pDMoGHfGIk44ewFvz+jBLZftTsPaWvoPbeDc616jW68CjRvFLy4dyVvTeqIaOOOKN9n7iFVZf4zMnHL2IsafvpQIeOvVblx98Qg2NeTkKZ15mPCymcmS+gI/IxlZXQP8ua03SRpG0owdSPKVTIyIa7c81G1j3ptd+fyJewNQUxPcNvllnnpo+6uw1tYGn/j32YzYby3r19RyxUfHsM/RK/j5V0by8X+fzV6H1zPpfwby4H8N4R8vfpvH79gZgG898iL1S+u4+sx9uPz+qdTk5G+6HP0HbuSkzyxmwgf3YWNDDZf95E3GfWw5j/xmQNahtZtqGUVt89cvIj4fESsj4qfA8cBZaVO1LY3ARRExGjgcOE/S6K0Ld9sac9RqFs7pwuL5XbIOZZvrO3ATI/ZbC0C3ngUGj1zHine68M7sbux5WD0A+xy9ghceTP5oF7zRjb2PXAlA7wGb6N67kbem9cwk9kpQ2yno3LVITW3QpVuRZYs6Zx1S+6qSJupmE5ykg96/AP2ATul6qyJiYURMSddXk0yxNKS9At8Wxv39Ch6/d4esw8jckrldmDO9B7sfuJohe6xjyh+Sa8Cff2AAyxcmf7jD917Li4/0p9AIS97uwluv9GTZgu3vHwPAskWd+c3EgfzymZe5ffI01tbXMuWJ3lmH1a4UpS1Za62JenUr2wI4rtSTSBoBHAg828K2CcAEgK50L/WQHa5TXZHDT1jJzd8ZnHUomdqwtobrz9mbMy6fTbdeBT77/Te47fLduO/a4Rx4/DJq65Lf4qM/vogFs7pzxd+NYcCQBkYdXE9NbQX8hmegZ59Gjjh+FZ/+wL6sqe/Ev93wF447ZRl/uqd/1qG1n2rvg4uIY9vjBJJ6AncBX4qI+hbOMxGYCNBb/SrmL+KQY+uZ9XJ3Vi6tyzqUzDRuEtefszdHnLKYsR9eBsDgkeu55LbpALzzZlde+lNSm6vtBGdc/t7gzLdO2Z+dd12/7YOuAAcetZpFczuzannyu/PUQzuw98Fr85PgKqT5WYoO7QJOZwC+C7gtIu7uyHO1t3EnreDxe/u1vWNORcDNl4xi0Mh1jP+XBe+W16cJv1iE+64bzrGffAeAhvU1NKxLfp1emdSXmtpgyB7bZ4JbPL8zex20li5di0Aw5gP1zJ2Vs/kpqqQPrsOebK/kIQ43ATMj4gcddZ6O0KVbgYOOqefarw3POpTMvPF8b56+eyeG7rWWr48fA8CpX5nDotndePTWQQAcPH4pR5+2CEgS39Wf2gfVwA4DNzLhh69nFXrmXpvagyce3IHrH5xBoSD+Mr07v789PyOoAGqnCS8l3Qz8HbA4IvZNy64A/gVYku52WUQ8mG67FDib5LkwX4yIh1s9fnTQpE2SjgKeAF7mvfk/3w20Jb3VLw6rPaFD4smDX7z1v1mHUPE+s9u4rEOoaM80Pkx9cflWdaB1GTYshl5wYUn7vnnJRS9ExNjNbZd0DMmlZ7e+L8GtiYir3rfvaJK5KA8lmZvyj8AeEbHZh2CVcquWSKYs3y0irpQ0HNg5Ip5r7X0R8SSVMCm7mbWr9hwhjYhJ6SBkKU4CfhURDcBsSbNIkt1mr8stpQ/uJ8ARwOnp69XAj0sMyMzyqPQpywdImtxsmVDiGb4gaZqkmyU1Xas1BJjbbJ95tHHpWSkJ7rCIOA/YABARK4CcXbVoZmUpfZBhaUSMbbZMLOHoNwC7kzzBbyGtX7LWqlIGGTZJqiUNV9KOVM0zdcysI3TkRbwRsejd80g/A+5PX87nr++DH5qWbVYpNbjrgHuAnSR9m2SqpP9fTsBmliORjKKWsmwJSYOavTwFeCVdvw/4hKQuknYFRgGtjgWU8lzU2yS9QDJlkoCTI8JPtjfbnrVTDU7SHSTzTQ6QNA+4HBgnaUx6lrdIH5cQEdMl3QnMILnX/bzWRlChtFHU4cA64HfNyyLi7S34PGaWB+03inp6C8U3tbL/t4Fvl3r8UvrgHuC9h890BXYFXgP2KfUkZpYvlXAjfSlKaaLu1/x1OpPI5zssIjOzdlL2rVoRMUXSYR0RjJlVibzU4CR9udnLGuAgYMFmdjezvIv2uxe1o5VSg+vVbL2RpE/uro4Jx8yqQh5qcOkFvr0i4uJtFI+ZVTiRg0EGSZ0iolHSB7ZlQGZWBao9wZFcIXwQMFXSfcCvgbVNG6ttAkszaycV8ryFUpTSB9cVWEbyDIam6+ECcIIz217lYJBhp3QE9RXeS2xNqiR/m1lHyEMNrhboScuTVlbJxzOzDlElGaC1BLcwIq7cZpGYWXWokAfKlKK1BOfpxs2sRXloon5wm0VhZtWl2hNcRCzfloGYWfXI061aZmbvyUkfnJnZ3xDV00HvBGdm5XMNzszyKg+jqGZmLXOCM7NcytmEl2Zmf801ODPLK/fBmVl+OcGVT106Uzt8eNZhVKzPHXNG1iFUvGNfnJl1CBVtxmmtPgi+ZK7BmVk+BVUz4WVN1gGYWXVpeuhMKUubx5JulrRY0ivNyvpJekTSG+nPHdJySbpO0ixJ09KH0LfKCc7MyhclLm37BTD+fWVfAx6NiFHAo+lrgA8Do9JlAnBDWwd3gjOzsimipKUtETEJeP/MRScBt6TrtwAnNyu/NRLPAH0lDWrt+E5wZlaeUmtvSX4bIGlys2VCCWcYGBEL0/V3gIHp+hBgbrP95qVlm+VBBjMrWxmjqEsjYuyWniciQtryMVvX4MysbCqWtmyhRU1Nz/Tn4rR8PjCs2X5D07LNcoIzs/K13yBDS+4DzkrXzwLubVZ+ZjqaejiwqllTtkVuoppZedrxyfaS7gDGkfTVzQMuB74D3CnpbGAOcFq6+4PAR4BZwDrgM20d3wnOzMrXTgkuIk7fzKa/eehVRARwXjnHd4Izs7I0XehbDZzgzKxsKlZHhnOCM7Py+KlaZpZnntHXzPLLNTgzyysPMphZPgVQwo30lcAJzszK5j44M8slXwdnZvkV4SaqmeWXa3Bmll9OcGaWV67BmVk+BVCojgznBGdmZXMNzszyy6OoZpZXrsGZWT55uiQzyysB8iCDmeVVKU+trwROcGZWHjdRq0td5wLf+9ET1NUVqa0Nnnx8MLf9fG8gOPNzMzn62PkUiuLB3+7KfXftnnW4majrXOC7P3k6/Y6KPPXYYG67aU8uvnwKo/ZaSWOhhtdn9OX67+5PobB9PG53wztixmVd2LhMSDD41E0M+2Qjb/6kjgV3daLzDkkW2O2LmxhwTAGANa+JV6/sQmGtQDD2V+up7ZLlp9gSvhcVSV2BSUCX9Dy/iYjLO+p8W2PTxhou/dJRbFjfidraIlf9+AkmPzuQ4busZsed1jPhkx8iQvTp25B1qJnZtLGGy84/4t3v6Ps/fYrJz+zE438YwlXfPBCAr3xzCif+/ds8eM+IbIPdRlQLoy7eSK/RRRrXwvMf70a/I5JENvxTmxj+6ca/2r/YCNMv7cro/2yg155FNq2EmiqtYlTLKGpH/qttAI6LiAOAMcD49GnUFUhsWJ/8pnXqVKS2UxECPnLybG6/ZU8iBMCqlVX3r7YdtfwdTf7zQNJuZ16f0ZcBO63PNMptqcuOQa/RycRonXpAj12LNCzSZvdf/nQtPfco0mvP5D11fZMkWZWaZhRpa8lYh/3/SB/SuiZ9WZcu2X/izaipCa792WMMHrKW+3+7G6/N7MegwWs55rj5HHn0Qlat7MxPr9ufBfN6Zh1qZmpqgmtvnsSgoWt54O4RvDZjh3e31dYWOXb8PCb+cN8MI8zO+vli9as19N6/yMqptcy7o46F99XRe58CIy/eSF0fWD8naZZOPacLG1eIgeML7PLZTVmHXr6onlHUDu0skVQraSqwGHgkIp7tyPNtjWJRnH/2cZx56onssdcKdtm1nrq6Ihs31nDBhHE8dP8IvvTVF7MOM1PFojj/0/+Ps04+nj32Xskuu9W/u+3zl7zMK1P7M/2l/hlGmI3GdfDKhV0Y9dWNdOoJQ0/bxBEPrufQ36yn847BrKs6AxAFWPViDaO/08DBt2xgyaO1LH+mSvsro8QlYx367UZEISLGAEOBQyX9zb93SRMkTZY0eWMh++bN2jWdmfbiAA4+bBFLl3Tj6UmDAXh60iB23X1VxtFVhrVr6pg2ZQAHH7YEgNM/+xp9+jZw43X7ZBzZtlfclCS3gR9tZKcPJf1vnQckTU/VwOB/bKT+laQd2mVg0PfgAp13gNpu0P/oAqtnVmcbVRElLW0eR3pL0suSpkqanJb1k/SIpDfSnzu0dZzN2Sb/PiJiJfAYML6FbRMjYmxEjO1c221bhPM3evdpoEfPjQB07lzgwLFLmDenF39+chD7H7gUgP3GLGX+3O23edq7bwM9eibNqc6dC4w5ZAlz5/TkhI/N4eDDlvC9bxz8bl/l9iICXr28M913C4af9d6AQsOS976HJY/W0mNk0ufW78gCa96oobA+GXBYObmWHrtXycMN3q99++COjYgxETE2ff014NGIGAU8mr7eIh05irojsCkiVkrqBhwPfLejzrc1+vXfwEWXTaGmNpCCJx4bwnN/3pnpL/fjkq+/wCmn/YX162q59nsHZh1qZvr1b+DLX3+RmppANfDko4N5/umB3DfpfhYv6sbVE58E4On/HcQdP98j42i3jVUv1vDO7+roMarIc6d2BZJLQhb9vhNrXq0BQbchRfb8RvLPs65PMro6+fRuIOh/dOO7l49UlQA6Ni+fBIxL128BHge+uiUHUnTQSIek/UmCqyWpKd4ZEVe29p4+XXeOI4af2SHx5IEaq/CPYRs75nczsw6hov34tKeYN33VVlW1+/QYHIePPqekff8w+Yo5wNJmRRMjYmLTC0mzgRUkafO/ImKipJUR0TfdLmBF0+tydeQo6jRg+63ymOVZseQq3NJmTc+WHBUR8yXtBDwi6dXmGyMipC2/6q5Kh3DMLDNNTdRSlrYOFTE//bkYuAc4FFgkaRBA+nPxlobqBGdmZWuPUVRJPST1aloHTgBeAe4Dzkp3Owu4d0vjrNIbRcwsU+3Tdz8QuCfpZqMTcHtEPCTpeeBOSWcDc4DTtvQETnBmVqb2uQ0rIt4EDmihfBnwwa0+AU5wZlYuP1XLzPLME16aWX45wZlZLgVQdIIzs1yqjLneSuEEZ2blc4Izs1wKoFAds6A4wZlZmQLCCc7M8spNVDPLJY+imlmuuQZnZrnlBGdmuRQBheqYXdoJzszK5xqcmeWWE5yZ5VN4FNXMciogfKGvmeWWb9Uys1yKKOexgZlygjOz8nmQwczyKlyDM7N88oSXZpZXvtnezPIqgPCtWmaWS+EJL80sx8JNVDPLrSqpwSkqaDRE0hJgTtZxNDMAWJp1EBXM30/bKu072iUidtyaA0h6iORzlWJpRIzfmvNtjYpKcJVG0uSIGJt1HJXK30/b/B1lqybrAMzMOooTnJnllhNc6yZmHUCF8/fTNn9HGXIfnJnllmtwZpZbTnBmlltOcC2QdLOkxZJeyTqWSiRpmKTHJM2QNF3SBVnHVEkkdZX0nKSX0u/nm1nHtL1yH1wLJB0DrAFujYh9s46n0kgaBAyKiCmSegEvACdHxIyMQ6sIkgT0iIg1kuqAJ4ELIuKZjEPb7rgG14KImAQszzqOShURCyNiSrq+GpgJDMk2qsoRiTXpy7p0cU0iA05wtlUkjQAOBJ7NOJSKIqlW0lRgMfBIRPj7yYATnG0xST2Bu4AvRUR91vFUkogoRMQYYChwqCR3dWTACc62SNq3dBdwW0TcnXU8lSoiVgKPAZndcL49c4KzsqWd6DcBMyPiB1nHU2kk7Sipb7reDTgeeDXToLZTTnAtkHQH8GdgT0nzJJ2ddUwV5gPAp4DjJE1Nl49kHVQFGQQ8Jmka8DxJH9z9Gce0XfJlImaWW67BmVluOcGZWW45wZlZbjnBmVluOcGZWW45wVURSYX0koxXJP1aUvetONYvJJ2art8oaXQr+46TdOQWnOMtSX/z9KXNlb9vnzWtbW9h/yskXVxujJZvTnDVZX1EjElnONkInNt8o6Qtes5tRHyujZlAxgFlJzizrDnBVa8ngJFp7eoJSfcBM9KbvL8v6XlJ0ySdA8ndB5Kul/SapD8COzUdSNLjksam6+MlTUnnMns0vZn+XODCtPZ4dHql/l3pOZ6X9IH0vf0l/SGdA+1GQG19CEm/lfRC+p4J79t2TVr+qKQd07LdJT2UvucJSXu1y7dpueQn21ehtKb2YeChtOggYN+ImJ0miVURcYikLsBTkv5AMuPHnsBoYCAwA7j5fcfdEfgZcEx6rH4RsVzST4E1EXFVut/twDUR8aSk4cDDwN7A5cCTEXGlpI8CpdwB8tn0HN2A5yXdFRHLgB7A5Ii4UNI30mN/geQhLudGxBuSDgN+Ahy3BV+jbQec4KpLt3QKHkhqcDeRNB2fi4jZafkJwP5N/WtAH2AUcAxwR0QUgAWS/tTC8Q8HJjUdKyI2Nyfeh4DRyS2pAPROZxY5BviH9L0PSFpRwmf6oqRT0vVhaazLgCLwP2n5fwN3p+c4Evh1s3N3KeEctp1ygqsu69MpeN6V/qGvbV4EnB8RD79vv/a8V7QGODwiNrQQS8kkjSNJlkdExDpJjwNdN7N7pOdd+f7vwGxz3AeXPw8D/5pOZ4SkPST1ACYBH0/76AYBx7bw3meAYyTtmr63X1q+GujVbL8/AOc3vZA0Jl2dBJyRln0Y2KGNWPsAK9LkthdJDbJJDdBUCz2DpOlbD8yW9E/pOSTpgDbOYdsxJ7j8uZGkf22Kkofm/BdJTf0e4I10260ks6X8lYhYAkwgaQ6+xHtNxN8BpzQNMgBfBMamgxgzeG8095skCXI6SVP17TZifQjoJGkm8B2SBNtkLclEka+Q9LFdmZb/M3B2Gt904KQSvhPbTnk2ETPLLdfgzCy3nODMLLec4Mwst5zgzCy3nODMLLec4Mwst5zgzCy3/g+ACvNQ+xOQNgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Results for highest accuracy model\n",
      "Trained ShallowConvNet on single session using train/test split and got accuracy of: 0.9104166666666667\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAgi0lEQVR4nO3deZgdVZ3/8fenO519IwshGwQMIYYtYICwRRbRgPoDnEFkZjTjoKAiAqKMMiqig6MjgjiDaDT8BpRhG0BQEGSJA1EJhBASSAgEkhAgC9n3pPve7/xxq6GJne57031T91Z/Xs9TT6pObd+up/PtU+dUnVJEYGaWRTVpB2BmVi5OcGaWWU5wZpZZTnBmlllOcGaWWZ3SDqCpAf1qY8TwurTDqFgvze6edghW5bayie2xTW05xodO7BGrVueK2vaZ2dseioiJbTlfW1RUghsxvI6nHhqedhgV60NDxqYdQuVTm/7vZt70/CNtPsbK1TmmPzSsqG3rBr8yoM0nbIOKSnBmVg2CXOTTDqIoTnBmVpIA8lTHCwJOcGZWsjyuwZlZBgVBvW9RzSyLAshVyS2qn4Mzs5LliaKmlkjqKukpSc9JekHSlUn5vpKmS1og6XZJnZPyLsnygmT9iNbidIIzs5IEkIsoamrFNuCkiDgUGAtMlDQe+AFwbUSMBNYA5ybbnwusScqvTbZrkROcmZUsX+TUkijYmCzWJVMAJwH/k5TfBJyRzJ+eLJOsP1lq+cFHJzgzK0kQ5IqcgAGSZjSZzmt6LEm1kmYBK4CHgVeAtRHRkGzyOjA0mR8KLAFI1q8D+rcUqzsZzKwkEVBffB/DyogYt/NjRQ4YK6kvcA8wus0BNuEanJmVSOSKnIoVEWuBqcDRQF9JjZWvYcAbyfwbwHCAZH0fYFVLx3WCM7OSBJCP4qaWSBqY1NyQ1A04BZhHIdH9bbLZJODeZP6+ZJlk/WPRyjcXfItqZiUrpXbWgsHATZJqKVS27oiI30maC9wm6V+BZ4EpyfZTgF9JWgCsBj7R2gmc4MysJIUHfdue4CJiNnBYM+WvAkc2U74VOKuUczjBmVlJAqiP6mjdcoIzs5IEIlclzfdOcGZWsnxUx8CiTnBmVpL2aoPbHZzgzKxEIuc2ODPLosKIvk5wZpZBEWJ71KYdRlGc4MysZHm3wZlZFhU6GXyLamaZ5E4GM8sodzKYWabl/KCvmWVRIOqjOlJHdURpZhXDnQxmllmBfItqZtnlToYKt32ruPRjI6nfXkOuAY7/8Do+9dVl3HvjAO755UCWLurCHXPm0Kd/7l37zZ/VjYs/OorLb1jE8R9Zl1L06arrkudHdy+grnNQ2yl44v6+/OrqvdIOq6Lc9OQLbNlYSz4PuQZx4WkHpB1Su4nAj4lIuhH4CLAiIg4q13l2VV2X4N/vfIVuPfI01MOXz9ifI05az4FHbOKoU9Zz2d+M/Kt9cjmYctUQ3vf+DSlEXDnqt4nLznoPWzfXUtspuOY3C3j6sV68OLNH2qFVlMvOGsn6NdmrQxQ6GarjVa1ypuH/AiaW8fhtIkG3HoVP0zbUi1y9kGDkwVvYa/j2Zve598aBHHfaOvoOaGh2fcchtm4u/IJ3qgtq64LWP2JuWZKjpqgpbWWLICIep/BhiIqVy8HnP3AAZx9yEIdN2MDowzfvdNuVS+v48+/78JFJK3djhJWrpib46cPzuX32Czz7eE/mP+va27uE+N6tr/Cfv5/PqX+frd+ZQOSjuClt2as/l6C2Fm54ZD4b19Vy5bkjWPRiV0aM3trstj+7Yijn/sub1KT/R6ki5PPiC6ccQI/eOa6YspB9DtjC4vnd0g6rYnz5zJGsWtaZPv3r+f5tr7BkQVeen94z7bDaTSXUzoqReoKTdB5wHsDeQ9MJp2efHIces5Gnp/baaYJ76blu/NvnRwCwbnUtTz3ai9paOObUjtnR0GjT+lqe+3NPjjhxgxNcE6uWdQZg3ao6/vT7PoweuzkzCa7wXdTqSHCpRxkRkyNiXESMG9h/9zVcrl1Vy8Z1hfNt2yJmPt6L4SO37XT7m6fP4+an5nLzU3M5/iPruPDfXu+wya1PvwZ69C70LnfumufwCRtZsqBrylFVji7dcnTrkXt7/n3v38Ci+Vm6Pu3/ZftySb0Gl5bVy+u4+qK9yedFPg8TPrqW8aes5ze/HMCdN+zJ6hV1fO4DoznypPVc8qMlaYdbUfoNqucr171GTQ3U1MDjv+3D9Ed6px1WxdhjYANXTFkIFJpBpv6mLzP+mJ3rU/hsYHX0oirK1P0l6VbgBGAAsBy4IiKmtLTPuEO7xlMPDS9LPFnwoSFj0w6h8in9WkMlm55/hPWxuk0XaeiBfeMLdxxX1LbfOOj+ZyJiXFvO1xZlq8FFxDnlOraZpataHvStjijNrGIUxoNTUVNLJA2XNFXSXEkvSLooKf+2pDckzUqm05rs83VJCyTNl/Sh1mLtsG1wZrar2m1E3wbg0oiYKakX8Iykh5N110bE1e86qzQG+ARwIDAEeETSqIh49/uUTTjBmVlJCo+JtL2tMyKWAkuT+Q2S5gFDW9jldOC2iNgGLJS0ADgS+MvOdvAtqpmVpPFd1GImYICkGU2m85o7pqQRwGHA9KToi5JmS7pR0h5J2VCg6SMNr9NyQnQNzsxKV8JwSStb60WV1BO4C7g4ItZLugH4LoXK4neBHwH/tCtxOsGZWUkKwyW1z+M4kuooJLdbIuLuwvFjeZP1vwB+lyy+ATR9jmxYUrZTvkU1s5K1x8v2kgRMAeZFxDVNygc32exM4Plk/j7gE5K6SNoX2B94qqVzuAZnZiUpjCbSLnWjY4FPAnMkzUrKLgfOkTSWwi3qIuB8gIh4QdIdwFwKPbAXtNSDCk5wZlaiwqtabU9wETENmn1Y7oEW9rkKuKrYczjBmVmJ2q0GV3ZOcGZWstbeUqgUTnBmVpL27EUtNyc4MyuZb1HNLJMav8lQDZzgzKwkATS4BmdmWeVbVDPLpgr5JGAxnODMrCSNA15WAyc4MyuZa3BmlkntNeDl7uAEZ2YlCURD3p0MZpZRboMzs2wK36KaWUa5Dc7MMs0JzswyKRA5dzKYWVa5k8HMMincyWBmWRZOcGaWTX7Z3swyzDW4XfDSnB5M3OfItMOoWL947bG0Q6h4nx3x/rRDyLwIyOWd4Mwso9yLamaZFPgW1cwyq3o6GarjcWQzqygRxU0tkTRc0lRJcyW9IOmipLyfpIclvZz8u0dSLkk/kbRA0mxJh7cWpxOcmZUsQkVNrWgALo2IMcB44AJJY4CvAY9GxP7Ao8kywKnA/sl0HnBDaydwgjOzkhR6UWuKmlo+TiyNiJnJ/AZgHjAUOB24KdnsJuCMZP504OYoeBLoK2lwS+dwG5yZlay1288mBkia0WR5ckRM3nEjSSOAw4DpwKCIWJqsWgYMSuaHAkua7PZ6UraUnXCCM7OSldCLujIixrW0gaSewF3AxRGxXnrn2BERkopPpzvwLaqZlSQorv2tmCQoqY5CcrslIu5Oipc33nom/65Iyt8AhjfZfVhStlNOcGZWsihyaokKVbUpwLyIuKbJqvuAScn8JODeJuWfSnpTxwPrmtzKNsu3qGZWmoBon1e1jgU+CcyRNCspuxz4PnCHpHOBxcDHk3UPAKcBC4DNwKdbO4ETnJmVrD3eZIiIabDTd75Obmb7AC4o5RxOcGZWshJ6UVO10wQn6T9o4TY6Ir5UlojMrKJl5V3UGS2sM7OOKoBqT3ARcVPTZUndI2Jz+UMys0pXLbeorT4mIuloSXOBF5PlQyX9tOyRmVmFEpEvbkpbMc/B/Rj4ELAKICKeAyaUMSYzq3Tt8SDcblBUL2pELGn6+gSQK084ZlbxIhudDI2WSDoGiOS1iosovPVvZh1VBdTOilHMLernKDxcNxR4ExhLiQ/bmVnWqMgpXa3W4CJiJfD3uyEWM6sW+bQDKE4xvaj7SfqtpLckrZB0r6T9dkdwZlaBGp+DK2ZKWTG3qP8N3AEMBoYAdwK3ljMoM6ts7fFNht2hmATXPSJ+FRENyfRroGu5AzOzClbtj4lI6pfM/l7S14DbKIR8NoVhS8yso6qA289itNTJ8AyFhNb4k5zfZF0AXy9XUGZW2XZ9EPHdq6V3UffdnYGYWZUIQQW8hlWMot5kkHQQMIYmbW8RcXO5gjKzClftNbhGkq4ATqCQ4B6g8PHVaYATnFlHVSUJrphe1L+lMHzwsoj4NHAo0KesUZlZZav2XtQmtkREXlKDpN4UPuE1vLWdqtnpn17GqeesRAp+f+tAfnPjXmmHtNutfrMzN14yivVvdQYFE/5uOR84902WzO3Bry9/D9s21dJ/2DY+85P5dOuVo2G7+NXXR7J4dk9UA5/49qsccPS6tH+M1PTo3cAlP3yNEQdsIQKuuXQf5s3smXZY7SMLA142MUNSX+AXFHpWNwJ/aW0nScMp3MYOonBJJkfEdbse6u6xz6jNnHrOSi76f++lvr6Gq25+iemP9mXp4o716F9NbXDWNxayz8Gb2Lqxlu9+eCxjjl/DTZeN5KxvLOSA8euZdvsgHvr5UM74yms8cWvhj8C3H36W9SvruO5TB/Ivv5tFTQf9MOXnr3ydGX/szb+evx+d6vJ06VYl7zYVqVp6UVv99YuIL0TE2oj4GXAKMCm5VW1NA3BpRIwBxgMXSBrTtnDLb++RW5k/qwfbttaSz4k503tx7MQ1aYe12/UdVM8+B28CoGvPHINHbmbtsi6sWNiNUUetB2DM8WuY+cAAAN58uRujj1kLQO8B9XTv3cDi2RmpsZSoe68cBx+1kQdv7Q9AQ30Nm9Zn7PtOVXKLutMEJ+nwHSegH9ApmW9RRCyNiJnJ/AYKQywNba/Ay2XRS9048IgN9OrbQJeuOY44cS0Dh2xPO6xUrVzShSUv9GDfwzYwZNRmZv2h8Az4jPsHsHppZwCGv3cTzz3cn1wDvPVaFxY/35PVb3ZJM+zU7DV8G+tWd+LSaxZz/YPzuPiHi+nSLVtDKCqKm9LW0p+VH7WwLoCTij2JpBHAYcD0ZtadB5wH0JXuxR6ybJYs6MadPxvM9349n62ba3jlhe7kc9XR3lAOWzfVcMP57+XsKxbSrVeOST98mduu2I/fXbc3h56yik51hd/iY89eztIF3fnXj4yl/9BtvOd966mprYDf8BTUdgpGHrSZ6785nPnP9uBzVy7h7AuWc/PVQ9IOrf1UextcRJzYHieQ1BO4C7g4ItY3c57JwGSA3jX9K+J/xEO3D+Sh2wcC8I9ffZ2VyzqnHFE6GurFDee/l6POXMHhp64CYPDILVxyywsALHu1K3MeK9TmajvB2VcsfHvf7595CIP23bL7g64AK5d25q2lnZn/bA8Apt2/Bx+/YFnKUbWjCrn9LEZZm4CTEYDvAm6JiLvLea721Kd/PQADh2zj2IlrmHpvv1b2yJ4IuOmr+zN45GY++Nk33y5fv7IOgHwe7v/J3rz/Hwr/cbdtqWHb5sKv09zH+1JTGwwZ1TET3Jq36lj5Zh3D9tsKwNjj1vPayxnrpKqSNriytXyq8BGHKcC8iLimXOcph2/+bAG99mggVy+u/9Y+2WsgLsKCp3vz5N17MnT0Jq6cOBaAj122mOULuzH15sEAHD5xJcd+fDkAG1bW8eNPHohqYI9B2zn3xy+lFXpFuP6bw/nn/1hEp855li3uwo8u3SftkNqV2qlTWNKNwEeAFRFxUFL2beCzwFvJZpdHxAPJuq8D51L4LsyXIuKhFo8fZRq0SdJxwBPAHN4Z//PtQJvTu6Z/jK+bWJZ4smDyK4+lHULF++yI96cdQkWbnvsD62N1mxrQugwfHsMuuqSobV/96qXPRMS4na2XNIHCo2c375DgNkbE1TtsO4bCWJRHUhib8hFgVETstAenmFe1RGHI8v0i4juS9gb2ioinWtovIqZRCYOym1m7as8e0oh4POmELMbpwG0RsQ1YKGkBhWS30+dyi2mD+ylwNHBOsrwBuL7IgMwsi4ofsnyApBlNpvOKPMMXJc2WdKOkPZKyocCSJtu8TiuPnhWT4I6KiAuArQARsQbomN2KZlZQfCfDyogY12SaXMTRbwDeQ+ELfktp+ZG1FhXTel4vqZYkXEkDqZpv6phZOZTzId6IWP72eaRfAL9LFt/g3e/BD0vKdqqYGtxPgHuAPSVdRWGopO+VErCZZUgUelGLmXaFpMFNFs8Enk/m7wM+IamLpH2B/YEW+wKK+S7qLZKeoTBkkoAzIsJftjfryNqpBifpVgrjTQ6Q9DpwBXCCpLHJWRaRfC4hIl6QdAcwl8K77he01IMKxfWi7g1sBn7btCwiXtuFn8fMsqD9elHPaaZ4SgvbXwVcVezxi2mDu593Pj7TFdgXmA8cWOxJzCxbKuFF+mIUc4t6cNPlZCSRL5QtIjOzdlLyO0gRMVPSUeUIxsyqRFZqcJK+3GSxBjgceHMnm5tZ1kX7vYtabsXU4Ho1mW+g0CZ3V3nCMbOqkIUaXPKAb6+I+MpuisfMKpzIQCeDpE4R0SDp2N0ZkJlVgWpPcBSeED4cmCXpPuBOYFPjymoawNLM2lGFfG+hGMW0wXUFVlH4BkPj83ABOMGZdVQZ6GTYM+lBfZ53ElujKsnfZlYOWajB1QI9aX7Qyir58cysLKokA7SU4JZGxHd2WyRmVh0q5IMyxWgpwXm4cTNrVhZuUU/ebVGYWXWp9gQXEat3ZyBmVj2y9KqWmdk7MtIGZ2b2V0T1NNA7wZlZ6VyDM7OsykIvqplZ85zgzCyTMjbgpZnZu7kGZ2ZZ5TY4M8suJ7jSqa6O2mGD0w6jYn3++Oa+kWtNnTpnTtohVLQXz2pol+O4Bmdm2RRUzYCXNWkHYGbVpfGjM8VMrR5LulHSCknPNynrJ+lhSS8n/+6RlEvSTyQtkDQ7+Qh9i5zgzKx0UeTUuv8CJu5Q9jXg0YjYH3g0WQY4Fdg/mc4Dbmjt4E5wZlYyRRQ1tSYiHgd2HLnodOCmZP4m4Iwm5TdHwZNAX0ktNto7wZlZaYqtvRXy2wBJM5pM5xVxhkERsTSZXwYMSuaHAkuabPd6UrZT7mQws5KV0Iu6MiLG7ep5IiKkXe+zdQ3OzEqmfHHTLlreeOuZ/LsiKX8DGN5ku2FJ2U45wZlZ6dqvk6E59wGTkvlJwL1Nyj+V9KaOB9Y1uZVtlm9Rzaw07fhle0m3AidQaKt7HbgC+D5wh6RzgcXAx5PNHwBOAxYAm4FPt3Z8JzgzK107JbiI2NnrOX/10auICOCCUo7vBGdmJWl80LcaOMGZWcmUr44M5wRnZqXxV7XMLMs8oq+ZZZdrcGaWVe5kMLNsCqCIF+krgROcmZXMbXBmlkl+Ds7MsivCt6hmll2uwZlZdjnBmVlWuQZnZtkUQK46MpwTnJmVzDU4M8su96KaWVa5Bmdm2eThkswsqwTInQxmllXFfLW+EjjBmVlpfItaXeo65/jB9X+iri5PbafgT1MHc8uU0Vz0tVmMHL0WKXhjSU+uveowtm7pmJesrnOOH9zw58I1qk2u0S8PYNDgzfzzd2fSq892FrzYhx9deRgNDR3jc7tblorZX+/OtlU1SDD8rG2M+OR2ABbd0pnXbu2CamDghHpGf2Ur29eKZy/uzrrnOzH0jO0c+I0tKf8Eu8rvoiKpK/A40CU5z/9ExBXlOl9b1G+v4fIvHcPWLZ2orc3zwxumMePJPZn8kwPZsrkOgM9c+Dwf/ZuF3Pnr/VOONh3122u4/ItHv3ONfv5nZvxlT84851V+c9u+PP7IUC64bDYf/OhrPHDPiLTD3S3UCUZftpU+Y3I0bII/ndWL/kc3sH1VDSseq+PYuzdQ2xm2rRIANZ2D/S/cysYFtWx4uTbl6NumWnpRy/mndhtwUkQcCowFJiZfo65Aertm1qlToRZH6O3kBkHnLrlq+aNVJk2vUVDbKQ8Bh7xvJdOmDgbg0QeGM37C8jSD3K26Dgz6jMkB0KkH9Nwvz7YVNbx2e2f2+8w2ajsXtuvSv/CL06k79HtfjprOaUXcjhpHFGltSlnZanDJR1o3Jot1yZT+T7wTNTXBdTf+L4OHbuL+u/dl/tw9ALj48mcZd/RylizqxZT/ODDlKNNVUxNc9/+fYPCwTdx/1wiWvtGDTRvryOcKfydXruhK/4FbU44yHZvfqGH9vFr6HNLAi1d3Y80znXjpuq7UdIHRX9lC34NzaYfYfqJ6elHL2lgiqVbSLGAF8HBETC/n+doinxcX/uMJTDrzg4was4Z99l0PwI+/dxifOv1DLFnUi+NPfjPlKNOVz4sLJ01g0ukfYNSYtQzbZ2PrO3UADZvg2Yu7896vbaGuJ0QO6teJo2/dyOhLtzDr0u6VUJlpX1HklLKyJriIyEXEWGAYcKSkg3bcRtJ5kmZImrE9v7mc4RRl08Y6Zs8cwPvGr3i7LJ8X//vIUI49YWmKkVWOwjXqz+iD19CjZz01tYXxqwfsuZVVb3VNObrdK18Pz17cgyEfrmevU+oB6Dooz6AP1CNB30NyUAPb1yjlSNuXIoqaWj2OtEjSHEmzJM1IyvpJeljSy8m/e+xqnLuluysi1gJTgYnNrJscEeMiYlznmu67I5y/0rvvNnr0LPxydu6cY+wRb/H6az0ZPLSxhhKMP24Zry/umUp8leBd16hLjrFHrGTJop7MmTmA404sJP6TT1vC9CcGpRnmbhUBc77VnR775dn3H7e9XT7o5HpWPVVo/dm0qIaoF533qIDqTHtq3za4EyNibESMS5a/BjwaEfsDjybLu6ScvagDgfqIWCupG3AK8INyna8t+vXfype/8Sw1NYFqYNpjQ3j6z4P4959Oo3uPBhAsXNCb6394SNqhpqZf/218+VuzCtdIMO2xwTz9p0EsWdiTy747k0+eP59XX+rDQ78dnnaou82ambW8eV9neo3KMe1jvQAYdfEWhp25nTnf7M4Tp/eipi445KrNKKnA/fGU3jRshHy9WP5YHUdM3kivkVXyBZdGAZQ35NOBE5L5m4A/Av+8KwdSlKlxQNIhFIKrpVBTvCMivtPSPn267BXHDPuHssSTCfkq+4+QglMemJN2CBXt2rOms+T59W26X+7TY0iMH3N+Udv+Yca3FwMrmxRNjojJjQuSFgJrKKTNn0fEZElrI6Jvsl7AmsblUpWzF3U2cFi5jm9mKSr+j+3KJreezTkuIt6QtCfwsKQXm66MiJB2/am7jvHIuZm1n8Zb1GKm1g4V8Uby7wrgHuBIYLmkwQDJvyt2foSWOcGZWcnaoxdVUg9JvRrngQ8CzwP3AZOSzSYB9+5qnB3zxUoza5v2absfBNxTaGajE/DfEfGgpKeBOySdCywGPr6rJ3CCM7MStc9rWBHxKnBoM+WrgJPbfAKc4MysVP6qlpllmQe8NLPscoIzs0wKIO8EZ2aZVBljvRXDCc7MSucEZ2aZFECuOt6LdoIzsxIFhBOcmWWVb1HNLJPci2pmmeYanJlllhOcmWVSBOSq4zOITnBmVjrX4Mwss5zgzCybwr2oZpZRAeEHfc0ss/yqlpllUkTVfKPXCc7MSudOBjPLqnANzsyyyQNemllW+WV7M8uqAMKvaplZJoUHvDSzDAvfoppZZlVJDU5RQb0hkt4CFqcdRxMDgJVpB1HBfH1aV2nXaJ+IGNiWA0h6kMLPVYyVETGxLedri4pKcJVG0oyIGJd2HJXK16d1vkbpqkk7ADOzcnGCM7PMcoJr2eS0A6hwvj6t8zVKkdvgzCyzXIMzs8xygjOzzHKCa4akGyWtkPR82rFUIknDJU2VNFfSC5IuSjumSiKpq6SnJD2XXJ8r046po3IbXDMkTQA2AjdHxEFpx1NpJA0GBkfETEm9gGeAMyJibsqhVQRJAnpExEZJdcA04KKIeDLl0Doc1+CaERGPA6vTjqNSRcTSiJiZzG8A5gFD042qckTBxmSxLplck0iBE5y1iaQRwGHA9JRDqSiSaiXNAlYAD0eEr08KnOBsl0nqCdwFXBwR69OOp5JERC4ixgLDgCMluakjBU5wtkuStqW7gFsi4u6046lUEbEWmAqk9sJ5R+YEZyVLGtGnAPMi4pq046k0kgZK6pvMdwNOAV5MNagOygmuGZJuBf4CHCDpdUnnph1ThTkW+CRwkqRZyXRa2kFVkMHAVEmzgacptMH9LuWYOiQ/JmJmmeUanJlllhOcmWWWE5yZZZYTnJlllhOcmWWWE1wVkZRLHsl4XtKdkrq34Vj/Jelvk/lfShrTwrYnSDpmF86xSNJffX1pZ+U7bLOxpfXNbP9tSV8pNUbLNie46rIlIsYmI5xsBz7XdKWkXfrObUR8ppWRQE4ASk5wZmlzgqteTwAjk9rVE5LuA+YmL3n/UNLTkmZLOh8Kbx9I+k9J8yU9AuzZeCBJf5Q0LpmfKGlmMpbZo8nL9J8DLklqj8cnT+rflZzjaUnHJvv2l/SHZAy0XwJq7YeQ9BtJzyT7nLfDumuT8kclDUzK3iPpwWSfJySNbperaZnkL9tXoaSmdirwYFJ0OHBQRCxMksS6iDhCUhfgT5L+QGHEjwOAMcAgYC5w4w7HHQj8ApiQHKtfRKyW9DNgY0RcnWz338C1ETFN0t7AQ8B7gSuAaRHxHUkfBop5A+SfknN0A56WdFdErAJ6ADMi4hJJ30qO/UUKH3H5XES8LOko4KfASbtwGa0DcIKrLt2SIXigUIObQuHW8amIWJiUfxA4pLF9DegD7A9MAG6NiBzwpqTHmjn+eODxxmNFxM7GxPsAMKbwSioAvZORRSYAH0v2vV/SmiJ+pi9JOjOZH57EugrIA7cn5b8G7k7OcQxwZ5NzdyniHNZBOcFVly3JEDxvS/6jb2paBFwYEQ/tsF17vitaA4yPiK3NxFI0SSdQSJZHR8RmSX8Euu5k80jOu3bHa2C2M26Dy56HgM8nwxkhaZSkHsDjwNlJG91g4MRm9n0SmCBp32Tffkn5BqBXk+3+AFzYuCBpbDL7OPB3SdmpwB6txNoHWJMkt9EUapCNaoDGWujfUbj1XQ8slHRWcg5JOrSVc1gH5gSXPb+k0L42U4WP5vycQk39HuDlZN3NFEZLeZeIeAs4j8Lt4HO8c4v4W+DMxk4G4EvAuKQTYy7v9OZeSSFBvkDhVvW1VmJ9EOgkaR7wfQoJttEmCgNFPk+hje07SfnfA+cm8b0AnF7ENbEOyqOJmFlmuQZnZpnlBGdmmeUEZ2aZ5QRnZpnlBGdmmeUEZ2aZ5QRnZpn1fxdxysCxikJoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################################\n",
    "# MULTI TRIAL SHALLOWCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Configure\n",
    "retrain_model = False\n",
    "tensorboard_name = \"ShallowCN_multisession_C0\"\n",
    "best_model_filename = f\"./saved_variables/6/ShallowConvNet/{tensorboard_name}\"\n",
    "\n",
    "\n",
    "# Train on train/test split of data from one session\n",
    "## Note: the model is forced to use GPU, if GPU is not available replace with what is available e.g. /cpu:0\n",
    "if (retrain_model): # Retrain or not\n",
    "    with tf.device('/gpu:0'):\n",
    "        keras_shallowcn_model.fit(\n",
    "            x= mne_fixed_window_epochs_train_data,\n",
    "            y= ohe_labels_train,\n",
    "            batch_size= 128, # Default: 32\n",
    "            epochs= 500, # Default: 500 (EEGNet paper)\n",
    "            verbose= 1, # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "            #callbacks= [tensorboard_callback(\"EEGNet_raw_signal_0.5s_100samps_50kernlen_02nr\")], # To be used for TF Board\n",
    "            callbacks= [tensorboard_callback(tensorboard_name),\n",
    "                        lowest_loss_model_save_callback(best_model_filename),\n",
    "                        highest_accuracy_model_save_callback(best_model_filename)],\n",
    "            validation_split= 0.3,\n",
    "            shuffle= True,\n",
    "            sample_weight= None, # Can be interesting due to time series\n",
    "            use_multiprocessing=True, # Done for faster speed\n",
    "            workers= 4 # Done for faster speed\n",
    "            )\n",
    "\n",
    "# Convert labels back to original\n",
    "y_test = ohe.inverse_transform(ohe_labels_test)\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"Results for lowest loss model\")\n",
    "keras_shallowcn_model = load_lowest_loss_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_shallowcn_model.predict(mne_fixed_window_epochs_test_data)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained ShallowConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"\\n\\nResults for highest accuracy model\")\n",
    "keras_shallowcn_model = load_highest_accuracy_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_shallowcn_model.predict(mne_fixed_window_epochs_test_data)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained ShallowConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Remove unused variables\n",
    "del accuracy\n",
    "del best_model_filename\n",
    "del retrain_model\n",
    "del tensorboard_name\n",
    "del y_pred\n",
    "del y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "59c7864b",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAUP\n",
    "####################################################\n",
    "\n",
    "# delete unused variables\n",
    "del keras_shallowcn_model\n",
    "del mne_fixed_window_epochs_test_data\n",
    "del mne_fixed_window_epochs_train_data\n",
    "del ohe\n",
    "del ohe_labels_test\n",
    "del ohe_labels_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6032b382",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## DeepConvNet\n",
    "\n",
    "With the ShallowConvNet far outperforming the best CSP approach but being worse in inter-session performance then EEGNet, we expect the DeepConvNet to perform better in the inter-session experiments.\n",
    "Again, the original implementation of the model works on 250Hz data.\n",
    "The defaults provided by the EEGModels library are adopted for 128Hz, we will further adapt it to 100Hz here.\n",
    "Contrary to the EEGNet implementation of EEGModels, the implementation of ShallowConvNet and DeepConvNet are not verified by the original authors, Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730)).\n",
    "However, the found results are in line with the paper, thus the implementation is believed to be correct.\n",
    "Once again, the functions provided in EEGModels are slightly altered to allow for parameter tuning.\n",
    "\n",
    "### Fixed window classification: Single trial | DeepConvNet | three class MI task | 100Hz input signal\n",
    "\n",
    "The scores are given for the best CSP approach as well as for the trained DL approaches.\n",
    "Scores for the DL approach are a combination of the best model based on validation loss / validation accuracy.\n",
    "TensorBoard was used to monitor the behaviour of the model over time, to ensure no overfitting tendencies occur and that enough epochs have happened to learn optimally.\n",
    "\n",
    "**The result on subject `C` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** | **DeepConvNet: test accuracy** |\n",
    "|----------------|------------------------------|---------------------------|-----------------------------------|--------------------------------|\n",
    "| 0              | 0.896                        | 0.944 / 0.934             | 0.951 / 0.955                     | 0.937 / 0.944                  |\n",
    "| 1              | 0.847                        | 0.931 / 0.913             | 0.937 / 0.937                     | 0.906 / 0.917                  |\n",
    "| 2              | 0.719                        | 0.892 / 0.878             | 0.868 / 0.871                     | 0.861 / 0.892                  |\n",
    "\n",
    "**The result on subject `B` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** | **DeepConvNet: test accuracy** |\n",
    "|----------------|------------------------------|---------------------------|-----------------------------------|--------------------------------|\n",
    "| 0              | 0.465                        | 0.569 / 0.587             | 0.496 / 0.542                     | 0.490 / 0.535                  |\n",
    "| 1              | 0.560                        | 0.701 / 0.684             | 0.649 / 0.601                     | 0.618 / 0.642                  |\n",
    "| 2              | 0.653                        | 0.816 / 0.792             | 0.694 / 0.712                     | 0.705 / 0.726                  |\n",
    "\n",
    "\n",
    "**The result on subject `E` are:**\n",
    "\n",
    "| **File index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** | **DeepConvNet: test accuracy** |\n",
    "|----------------|------------------------------|---------------------------|-----------------------------------|--------------------------------|\n",
    "| 0              | 0.701                        | 0.792 / 0.785             | 0.802 / 0.806                     | 0.729 / 0.771                  |\n",
    "| 1              | 0.580                        | 0.795 / 0.799             | 0.826 / 0.823                     | 0.771 / 0.781                  |\n",
    "| 2              | 0.805                        | 0.864 / 0.864             | 0.916 / 0.923                     | 0.840 / 0.857                  |\n",
    "\n",
    "With EEGNet outperforming DeepConvNet or being very close, EEGNet is preferred over DeepConvNet.\n",
    "In fact, the single session performance of ShallowConvNet is better then that of DeepConvNet.\n",
    "This is mostly due to EEGNet allowing for more control to combat overfitting tendencies, which DeepConvNet suffers from.\n",
    "Thus DeepConvNet is not promosing over EEGNet but ShallowConvNet is still very interesting as it allows for explainibility as further discussed in the paper by Schirrmeister et al ([2017](https://doi.org/10.1002/hbm.23730)).\n",
    "ShallowConvNet is also faster in training and is likely more sample efficient.\n",
    "In what follows next we study if the Deep variant of ConvNet does indeed perform better inter-session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d0a235ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 0 ... 666799  =      0.000 ...  3333.995 secs...\n",
      "Using data from preloaded Raw for 960 events and 801 original time points ...\n",
      "0 bad epochs dropped\n",
      "Loaded fixed window binary epochs:\n",
      "\n",
      "Extracted labels from epochs: [1 2 1 3 1 3 3 2 3 2]\n",
      "One Hot Encoded labels: [[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]]\n",
      "Labels match before and after the One Hot Encoding: True\n",
      "Shape of data (epochs, channels, samples): (960, 21, 100)\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# PREPPING THE DATA\n",
    "####################################################\n",
    "\n",
    "# Specify raw to use\n",
    "mne_raw = load_mne_raw(subject= \"C\", index= 0)\n",
    "\n",
    "# Get the epoch from the RAW limited to MI tasks\n",
    "# Include period before and after to enable filtering possibilities\n",
    "mne_fixed_window_epochs = CLA_dataset.get_usefull_epochs_from_raw(mne_raw, start_offset=-1.5, end_offset=1.5)['task/neutral', 'task/left', 'task/right']\n",
    "\n",
    "# Load the epochs, we don't need filtering since this is what we want to learn!\n",
    "mne_fixed_window_epochs.load_data()\n",
    "print(f\"Loaded fixed window binary epochs:\\n\")\n",
    "\n",
    "# Labels: should be one hot encoded!\n",
    "labels = mne_fixed_window_epochs.events[:, -1]\n",
    "print(f\"Extracted labels from epochs: {labels[:10]}\")\n",
    "\n",
    "# Go to 2D representation\n",
    "labels = labels.reshape(-1, 1)\n",
    "\n",
    "# One Hot Encode the labels\n",
    "ohe = OneHotEncoder()\n",
    "ohe_labels = ohe.fit_transform(labels).toarray()\n",
    "print(f\"One Hot Encoded labels: {ohe_labels[:10]}\")\n",
    "\n",
    "# Show ohe labels\n",
    "np.shape(ohe_labels)\n",
    "ohe_labels[:10]\n",
    "\n",
    "# Validate OHE\n",
    "print(f\"Labels match before and after the One Hot Encoding: {np.array_equal(ohe.inverse_transform(ohe_labels), labels)}\")\n",
    "\n",
    "# Get effective data (half a second)\n",
    "mne_fixed_window_epochs_data = mne_fixed_window_epochs.get_data(tmin=0.2, tmax=0.7)\n",
    "\n",
    "# Fix scaling sensitivity as MNE stores as data * 10e-6\n",
    "mne_fixed_window_epochs_data = mne_fixed_window_epochs_data * 1000000\n",
    "\n",
    "# Delete unused variables\n",
    "del mne_fixed_window_epochs\n",
    "print(f\"Shape of data (epochs, channels, samples): {np.shape(mne_fixed_window_epochs_data)}\")\n",
    "\n",
    "# Remove unused variables\n",
    "del labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7fe5bf2e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_4\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_5 (InputLayer)        [(None, 21, 100, 1)]      0         \n",
      "                                                                 \n",
      " conv2d_6 (Conv2D)           (None, 21, 97, 25)        125       \n",
      "                                                                 \n",
      " conv2d_7 (Conv2D)           (None, 1, 97, 25)         13150     \n",
      "                                                                 \n",
      " batch_normalization_8 (Batc  (None, 1, 97, 25)        100       \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_10 (Activation)  (None, 1, 97, 25)         0         \n",
      "                                                                 \n",
      " max_pooling2d (MaxPooling2D  (None, 1, 48, 25)        0         \n",
      " )                                                               \n",
      "                                                                 \n",
      " dropout_2 (Dropout)         (None, 1, 48, 25)         0         \n",
      "                                                                 \n",
      " conv2d_8 (Conv2D)           (None, 1, 44, 50)         6300      \n",
      "                                                                 \n",
      " batch_normalization_9 (Batc  (None, 1, 44, 50)        200       \n",
      " hNormalization)                                                 \n",
      "                                                                 \n",
      " activation_11 (Activation)  (None, 1, 44, 50)         0         \n",
      "                                                                 \n",
      " max_pooling2d_1 (MaxPooling  (None, 1, 22, 50)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_3 (Dropout)         (None, 1, 22, 50)         0         \n",
      "                                                                 \n",
      " conv2d_9 (Conv2D)           (None, 1, 18, 100)        25100     \n",
      "                                                                 \n",
      " batch_normalization_10 (Bat  (None, 1, 18, 100)       400       \n",
      " chNormalization)                                                \n",
      "                                                                 \n",
      " activation_12 (Activation)  (None, 1, 18, 100)        0         \n",
      "                                                                 \n",
      " max_pooling2d_2 (MaxPooling  (None, 1, 9, 100)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_4 (Dropout)         (None, 1, 9, 100)         0         \n",
      "                                                                 \n",
      " conv2d_10 (Conv2D)          (None, 1, 5, 200)         100200    \n",
      "                                                                 \n",
      " batch_normalization_11 (Bat  (None, 1, 5, 200)        800       \n",
      " chNormalization)                                                \n",
      "                                                                 \n",
      " activation_13 (Activation)  (None, 1, 5, 200)         0         \n",
      "                                                                 \n",
      " max_pooling2d_3 (MaxPooling  (None, 1, 2, 200)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_5 (Dropout)         (None, 1, 2, 200)         0         \n",
      "                                                                 \n",
      " flatten_2 (Flatten)         (None, 400)               0         \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 3)                 1203      \n",
      "                                                                 \n",
      " activation_14 (Activation)  (None, 3)                 0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 147,578\n",
      "Trainable params: 146,828\n",
      "Non-trainable params: 750\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE DEEPCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Create the TensorFlow Keras model\n",
    "keras_deepcn_model = DeepConvNet(\n",
    "    nb_classes = 3, # int, number of classes to classify. \n",
    "    Chans = 21, # number of channels in the EEG data. \n",
    "    Samples = 100, # number of time points in the EEG data. (default: 2s 250, paper: 1s 250)\n",
    "    dropoutRate = 0.6, # dropout fraction. (default: 0.5)\n",
    "    conv_filters = 4, # Conv2D kernel size (default: 5, paper: 10)\n",
    "    strides = 2, # Stride size for average pooling layer (default: 2, paper: 3)\n",
    "    pool_size = 2 # Pool size for average pooling layer (default: 2, paper: 3)\n",
    "    )\n",
    "\n",
    "# Compile the model so it can be fitted\n",
    "# Loss and optimizer from EEGNet paper\n",
    "keras_deepcn_model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics=[\"accuracy\"])\n",
    "\n",
    "# Show summary of the model\n",
    "keras_deepcn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "10905bd4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for lowest loss model\n",
      "Trained DeepConvNet on single session using train/test split and got accuracy of: 0.9375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbNUlEQVR4nO3deZhcVZ3/8fenl+z7QkhCIgFiMgEhwciqeRAR4vIbwMdRXJDfDA4uoIg6I6jzw2HEkXFFBSWCirKvAyiQIAQRlJCFkIQESCCCgcTsZE+6q7+/P+o2NDHpqkqq+t7b+bye5z6pe2/Vud+udL4559xzzlVEYGaWZ3VpB2BmtrecyMws95zIzCz3nMjMLPecyMws9xrSDqCtQQPq48ARjWmHkVnPzeuRdgjZJ6UdQaZti83siG179SWd8s6esWZtoaz3zp63fWpETN6b65UjU4nswBGNPDF1RNphZNYpB7w17RAyT42Z+pXOnMe337fXZaxeW2DG1APKem/j0OcH7fUFy+C/dTOrUFCIlrSDeAMnMjOrSAAtZGsgvROZmVWsBdfIzCzHgqDJTUszy7MACm5amlneuY/MzHItgELGVs1xIjOzimWrh8yJzMwqFIT7yMws3yKgKVt5zInMzColCmRrTqsTmZlVJIAW18jMLO9cIzOzXCsOiHUiM7McC6ApsrUmqxOZmVUkEIWMLS7tRGZmFWsJNy3NLMfcR2ZmnYAouI/MzPKsuEKsE5mZ5ViE2BH1aYfxBk5kZlaxFveRmVmeFTv73bQ0s1xzZ7+Z5Zw7+82sUyh4QKyZ5VkgmiJbqSNb0ZhZ5mWxsz9b0ZhZ5gWiEOVtpUi6QNLTkhZIulFSN0mjJM2QtETSzZK6lCrHiczMKtZCXVlbeyQNBz4PTIyIw4B64AzgMuAHEXEIsA44u1Q8TmSJO68exDnvHMO/njCGO34++A3nbvvZYE4ZNp5X12RrNHNavvjdF7l57jyu+v3CtEPJpEFDt3PZDYu4ato8rpo6n1P/74q0Q6qqCChEXVlbGRqA7pIagB7AcuBE4Lbk/LXAaaUKqVkik/QLSSslLajVNarlL890477rB/Kj3z3Hz37/LDMe6MPLS4u12ZUvNzLnD73Zb/iOlKPMjmm3DuBrHz8k7TAyq6VZ/PzSkXzq5MP5wgfG8X8+8TdGHrI17bCqptjZX1/WBgySNKvNds5r5US8DHwXeIliAnsVmA2sj4jm5G3LgOGlYqpljexXwOQall81Ly3uytgJW+jWI6hvgMOP3cRj9/YD4KpvDOfsr7+CsnW3OVULZvRm43rXTndn7aouLHm6JwBbN9fz1yXdGbh/5/qPsEBdWRuwOiImttmmtJYhqT9wKjAKGAb0ZA9zRs0SWUQ8AqytVfnVdODYbSx4oicb1tazbYuY+VAfVr3SyJ/u78Og/Zs4+NBtaYdoOTVk+HYOHreFZ+f2SjuUqglES5S3lXASsDQiVkVEE3AHcDzQL2lqAhwAvFyqIA+/AEaO3s6HPruSiz5yMN16tHDQoVtp2iFu+vEQ/vvG59MOz3KqW48CX//pYq76r5Fs2dS5arBVGn7xEnCMpB7AVuBdwCxgOvBB4CbgLOCuUgWl3tkv6ZzW9vOqNYXU4pj80bVcMfU5vnfnEnr1LfCmMdtY8VIXPnPSWD5x1DhWLW/k3FPGsHalc7+VVt/Qwn/8dDHT7xrIY1MHpB1OVRWfa1lX1tZuOREzKHbqzwHmU8xHU4CvAF+UtAQYCFxTKqbU/1UmbeYpABOP6JbaYz/Xr26g36BmVi5r5LF7+3L5bxdz+idXv3b+E0eN48f3PUvfgeklW8uL4ILLlvLSku7ccc3QtIOpgeo9aTwiLgYu3unwC8BRlZSTeiLLiks+eSAb1zVQ3xic961l9OrrhLU7F/5kKYcfu5G+A5q5buZ8fvO9oUy9aVDaYWXGoRM3cdIH1rD0me5c8bviTftffecAZj7cL93AqqT4OLhsNZVrlsgk3QicQPH26zLg4ogoWUVMy/f/d0m753/9hMdMtfr2eaPSDiHTnp7Vm8mjKqpQ5EqESjYbO1rNEllEfKRWZZtZurwemZnlWnE9smwNrHQiM7MKeYVYM8u54vAL18jMLMda51pmiROZmVXMa/abWa4Vl/Fx09LMcs59ZGaWa8XVL9y0NLMcK05RciIzs1xzjczMOgGP7DezXPNdSzPrFNy0NLNca12zP0ucyMysIgE0u0ZmZnnnpqWZ5Vt5j3rrUE5kZlYRL6xoZp2Ca2RmlmteWNHMci8QzS3u7DeznHMfmZnlW7hpaWY55z4yM+sUnMjMLNcCUXBnv5nlnTv7zSzXwp39ZtYZhBOZmeWbJ42bWSfgGlk7npvXg1OGjU87jMy6bOmf0g4h87729tPTDiHT9Le9/ycfAYUWJzIzyznftTSzXAuy17TM1qg2M8uBYmd/OVvJkqR+km6T9IykRZKOlTRA0gOSFid/9i9VjhOZmVUsorytDJcD90fEWOAIYBFwIfBgRIwGHkz22+VEZmYVi1BZW3sk9QUmAdcUy4wdEbEeOBW4NnnbtcBppeJxH5mZVaR417LsOtAgSbPa7E+JiCnJ61HAKuCXko4AZgPnA0MiYnnynhXAkFIXcSIzs4qV2WwEWB0RE3dzrgE4EvhcRMyQdDk7NSMjIiSVvJqblmZWsWo0LYFlwLKImJHs30Yxsf1N0lCA5M+VpQpyIjOzigTlJbFSiSwiVgB/lTQmOfQuYCFwN3BWcuws4K5SMblpaWYVK79lWdLngOsldQFeAP6ZYgXrFklnAy8CHypViBOZmVUmIKo0RSki5gK76kN7VyXlOJGZWcWyNrLficzMKlbBXcsOsdtEJunHtNMUjojP1yQiM8u0LM61bK9GNqudc2a2rwogL4ksIq5tuy+pR0RsqX1IZpZ1WWtalhxHlsxGXwg8k+wfIenKmkdmZhkloqW8raOUMyD2h8ApwBqAiHiK4kRPM9tXRZlbBynrrmVE/FV6Q3Yt1CYcM8u8yFdnf6u/SjoOCEmNFGenL6ptWGaWaXnrIwM+DZwLDAdeAcYn+2a2z1KZW8coWSOLiNXAxzogFjPLi5a0A3ijcu5aHiTpHkmrJK2UdJekgzoiODPLoNZxZOVsHaScpuUNwC3AUGAYcCtwYy2DMrNsq+Ka/VVRTiLrERG/iYjmZLsO6FbrwMwsw/Iy/ELSgOTlfZIuBG6iGNqHgXs7IDYzy6ocDb+YTTFxtUb8qTbnArioVkGZWbaVXkW/Y7U313JURwZiZjkRgg6cflSOskb2SzoMGEebvrGI+HWtgjKzjMtLjayVpIuBEygmsnuB9wCPAk5kZvuqjCWycu5afpDi+tkrIuKfKT7WvG9NozKzbMvLXcs2tkZEi6RmSX0oPmNuRI3jStXEEzbw6f96hfq64L4bB3DLT0o+6Hif8Mdr9ueJmwcjwf5jtvBP33mBqz8+lu2b6wHYtKaREUds4qwpi1OONF3D37SJC78197X9/Ydt4bopo7nrxk7S7ZynhRXbmCWpH/BzincyNwF/LvUhSSMoNj+HUPzRp0TE5XseaseoqwvO/dbLXHTGQaxe3siP713M41P78tLifXvo3KsrGnnsV0P40gPzaOwWXHfuITx1z0A+c+vr6wf85jOjGXfSuhSjzIaXX+zF5z72dqD4+/Trex/iT9P3Tzmq6sraXcuSTcuI+GxErI+InwHvBs5KmpilNANfiohxwDHAuZLG7V24tTdmwhZe+UsXVrzUleamOh6+qx/HnvJq2mFlQktBNG2ro9AMTdvq6LNf02vntm2s5/k/9eHQk53I2jribatZvqwHq1Z0TzuU6spL01LSke2di4g57RUcEcuB5cnrjZIWUVxBY+EextohBu7fxKpXury2v3p5I2OP9ArfffdvYtK/Lue/j59AY7cWRr/jVd486fUE//S0/hx83Aa69fZSdW1NOnk5f5g6LO0wqi5rNbL2mpbfa+dcACeWexFJBwITgBm7OHcOcA5AN3qUW6R1sC2v1rPwgf585ZG5dO9T4LpzD2HOnQM58vQ1AMy9ZyBHfXhlylFmS0NDC0dPWsm1V4xJO5Tqy0sfWUS8sxoXkNQLuB34QkRs2MV1pgBTAPpoQOp5fs2KRgYP2/Ha/qChTaxe3phiRNmw5NG+9B+xnV4DmwE47JR1vDinN0eevobNaxtY9lRPPnHV+nSDzJiJx63i+Wf6sH5t17RDqa4ObjaWo5zhF3ssWVH2duD6iLijlteqlmfn9mD4qB0MGbGdhsYWTjh1PY9P82iTfsO289KTvdixtY4IWPKnPux38FYA5t83gLEnrqexa8Z+u1M26ZTl/GFa52tWAvnpI9tbKi7yfw2wKCK+X6vrVFtLQVzxteF864YXqKuHaTcN4MXn9u07lgAjJ2zmLe9Zy4/efxh1DcGwcVs4+iPFpuRT9wzkhM+8knKE2dK1WzMTjlrNT751aNqh1IQytrBizRIZcDxwJjBf0tzk2FcjIvMrZ8x8qA8zH+qTdhiZc/IFL3PyBS//3fFP3eRHOOxs+7YGPvLuk9IOo3YyVvkuZ4qSKC51fVBEXCJpJLB/RDzR3uci4lE6ctFuM+sQiuzdtSynj+xK4FjgI8n+RuCKmkVkZtmXsaWuy2laHh0RR0p6EiAi1knqUupDZtaJZaxGVk4ia5JUTxK6pMFk7hkqZtaRsta0LCeR/Qi4E9hP0qUUV8P4ek2jMrPsihzetYyI6yXNpriUj4DTIsK3qcz2ZXmrkSV3KbcA97Q9FhEv1TIwM8uwvCUy4He8/hCSbsAo4Fmgc470M7OSstZHVs4yPm+JiMOTP0cDR1HGemRmZuWQVC/pSUm/TfZHSZohaYmkm8sZJVHxXMtk+Z6j9yBeM+ssqjvX8nygbb/7ZcAPIuIQYB1wdqkCyukj+2Kb3TrgSMAT68z2VVW8aynpAOB9wKXAF5OZRCcCH03eci3wDeCn7ZVTTh9Z7zavmyn2md1eYbxm1pmUX9saJGlWm/0pydJdrX4I/Duv55mBwPqIaE72l1FckLVd7SayZCBs74j4crlRm1nnJirq7F8dERN3WY70fmBlRMyWdMLexNTeUtcNEdEs6fi9uYCZdULVuWt5PPCPkt5LcUREH+ByoF9r/gEOAP5+yZWdtNfZ37q6xVxJd0s6U9IHWre9/AHMLK/i9RUwSm3tFhNxUUQcEBEHAmcAD0XEx4DpFGcQAZwF3FUqpHL6yLoBayh2wLWOJwsgFyu+mlkN1HaK0leAmyR9E3iS4gKt7Wovke2X3LFcwOsJrFXGhsOZWUeq9oDYiHgYeDh5/QLF8aplay+R1QO92PXiiE5kZvuyjGWA9hLZ8oi4pMMiMbN8yOBTlNpLZF6m2sx2KWtzLdtLZO/qsCjMLF/yksgiYm1HBmJm+ZG7hRXNzN4gZ31kZmZ/R2SvA92JzMwq5xqZmeVdnu5ampntmhOZmeVaHh8HZ2b2d1wjM7O8cx+ZmeWfE9nuqaGe+n4D0g4jsy467MS0Q8i8KxfeknYImXba+6ozYcc1MjPLt6DWCytWzInMzCpS4cNHOoQTmZlVzonMzPJOka1M5kRmZpXx6hdm1hm4j8zMcs9TlMws/1wjM7NcK+Mp4h3NiczMKudEZmZ55gGxZtYpqCVbmcyJzMwq43FkZtYZePiFmeWfa2Rmlnfu7DezfAvAk8bNLO/cR2ZmueZxZGaWfxFuWppZ/rlGZmb5l7FEVpd2AGaWP4rytnbLkEZImi5poaSnJZ2fHB8g6QFJi5M/+5eKx4nMzCoTQCHK29rXDHwpIsYBxwDnShoHXAg8GBGjgQeT/XY5kZlZxapRI4uI5RExJ3m9EVgEDAdOBa5N3nYtcFqpeNxHZmaVq/JdS0kHAhOAGcCQiFienFoBDCn1eScyM6tYBXctB0ma1WZ/SkRMeUNZUi/gduALEbFB0mvnIiKk0ldzIjOzylS2jM/qiJi4u5OSGikmsesj4o7k8N8kDY2I5ZKGAitLXcR9ZGZWEQEqRFlbu+UUq17XAIsi4vttTt0NnJW8Pgu4q1RMrpGZWcWq9KTx44EzgfmS5ibHvgp8G7hF0tnAi8CHShXkRGZmlanSCrER8SjFCt6uvKuSspzIduGX9/+ZrVvqKRRES0Gcf8Zum/j7pMYuLXznhgU0dmmhviF49P6BXPejkWmHlbrfXz2Mx24aggTDxm7hrO88x/Oz+3D7paMoNImRb9nEmf+zmPrc/6vbh+ZaSuoGPAJ0Ta5zW0RcXKvrVduF/zKeDeu7pB1GJjXtEBd+4lC2bamnvqGF7960gFmP9OeZub3TDi0161Z0Yfovh3Hxg3Po0q2FKZ8dwxN37cdvfzCSL9wwnyEHbePu743k8duGcPwZf0s73L2WtbmWtezs3w6cGBFHAOOByZKOqeH1rMOIbVvqAWhoCBoaImv/QaeipSCattVRaIamrfV07VGgvrGFIQdtA+Af3rGeOfcNTDnKKmldAaPU1kFqlsiiaFOy25hsufh1j4BvXvUUl988k8kffCXtcDKpri74yd1zufHxmTz5WF+efWrfrY0B9N9/Byed8zJfPfZtfOVtR9OtdzNvff9qWgrixXm9AJhz7yDWLe+acqRVENW5a1lNNW2tS6oHZgOHAFdExIxaXq9a/u2sI1mzsit9B+zg0ilzWba0Bwtm90s7rExpaRHn/eN4evZu5j+ufIY3jd7Mi4t7ph1Waja/Ws+8aQP45qMz6dGnwJTPjuWJOwfzyR8/y62XjKJpRx3j3rGOuvpc/F9eWsZ+jJqOI4uIQkSMBw4AjpJ02M7vkXSOpFmSZu1o2VbLcMq2ZmXxf81X13bhzw8O5s2HbUg5ouzavLGBeTP6MnHS+rRDSdUzj/Zj4Iht9B7YTH1jMGHyGp6f3YeD3rqRL982n4vuforRR29gv1Fb0w61KhRR1tZROmRAbESsB6YDk3dxbkpETIyIiV3qunVEOO3q2r1A9x7Nr72ecNxaXlyy79Y0dqXvgCZ69i5+R126Fphw3Hr++kL3lKNK14Bh21n6ZG92bK0jAp55rC9DD9nChtWNADRtF1N/egCTPrYi5UirJGN9ZLW8azkYaIqI9ZK6A+8GLqvV9aql/8AdfP2H8wGorw8evncIsx/rJB20VdJ/8A6+/D9LqKsLVBf88b5BPDF9QNphpWrUhE0c+d41XPq+8dTXByMO3czbP7qCu7/7JuY/OIAImPTxFYw9/tW0Q917AWTs4SOKGmVNSYdTXIKjnmLN75aIuKS9z/RtHBzH9vtATeLpDGLb9rRDyLwrF05NO4RMO+19q5k/r2l3g1DL0rfnsDhm3KfKeu+0Wd+Y3d5cy2qpWY0sIuZRXJbDzDqblmxVyXI/xtjMOlgGm5ZOZGZWsY68I1kOJzIzq5wTmZnl2z40adzMOqnWpyhliBOZmVXMfWRmln9OZGaWawG0OJGZWa65s9/MOgMnMjPLtQAK2Rra70RmZhUKCCcyM8s7Ny3NLNd819LMOgXXyMws95zIzCzXIqBQSDuKN3AiM7PKuUZmZrnnRGZm+Ra+a2lmORcQHhBrZrnnKUpmlmsRfhycmXUC7uw3s7wL18jMLN+8sKKZ5Z0njZtZ3gUQGZuiVJd2AGaWM5EsrFjOVoKkyZKelbRE0oV7GpJrZGZWsahC01JSPXAF8G5gGTBT0t0RsbDSslwjM7PKVadGdhSwJCJeiIgdwE3AqXsSjiJDdx8krQJeTDuONgYBq9MOIsP8/ZSWte/oTRExeG8KkHQ/xZ+rHN2AbW32p0TElKScDwKTI+KTyf6ZwNERcV6lMWWqabm3X3C1SZoVERPTjiOr/P2U1hm/o4iYnHYMO3PT0szS8jIwos3+AcmxijmRmVlaZgKjJY2S1AU4A7h7TwrKVNMyg6akHUDG+fspzd/RbkREs6TzgKlAPfCLiHh6T8rKVGe/mdmecNPSzHLPiczMcs+JbBck/ULSSkkL0o4liySNkDRd0kJJT0s6P+2YskRSN0lPSHoq+X7+M+2YOjv3ke2CpEnAJuDXEXFY2vFkjaShwNCImCOpNzAbOG1PppZ0RpIE9IyITZIagUeB8yPi8ZRD67RcI9uFiHgEWJt2HFkVEcsjYk7yeiOwCBieblTZEUWbkt3GZHONoYacyGyvSDoQmADMSDmUTJFUL2kusBJ4ICL8/dSQE5ntMUm9gNuBL0TEhrTjyZKIKETEeIqj1Y+S5C6KGnIisz2S9P3cDlwfEXekHU9WRcR6YDqQufmJnYkTmVUs6cy+BlgUEd9PO56skTRYUr/kdXeK6209k2pQnZwT2S5IuhH4MzBG0jJJZ6cdU8YcD5wJnChpbrK9N+2gMmQoMF3SPIrzCR+IiN+mHFOn5uEXZpZ7rpGZWe45kZlZ7jmRmVnuOZGZWe45kZlZ7jmR5YikQjLUYYGkWyX12IuyfpU8xQZJV0sa1857T5B03B5c4y+S/u5pO7s7vtN7NrV3fhfv/4akL1cao3UOTmT5sjUixicrcuwAPt32pKQ9Wro8Ij5ZYuWKE4CKE5lZR3Eiy68/AocktaU/SrobWJhMVv6OpJmS5kn6FBRH40v6SfJ4+t8D+7UWJOlhSROT15MlzUnW0nowmRT+aeCCpDb4jmTk+u3JNWZKOj757EBJ05I1uK4GVOqHkPS/kmYnnzlnp3M/SI4/KGlwcuxgSfcnn/mjpLFV+TYt1/zwkRxKal7vAe5PDh0JHBYRS5Nk8GpEvE1SV+AxSdMorlAxBhgHDAEWAr/YqdzBwM+BSUlZAyJiraSfAZsi4rvJ+24AfhARj0oaSfHhEf8AXAw8GhGXSHofUM6MiH9JrtEdmCnp9ohYA/QEZkXEBZL+X1L2eRQf5vHpiFgs6WjgSuDEPfgarRNxIsuX7snSMFCskV1Dscn3REQsTY6fDBze2v8F9AVGA5OAGyOiALwi6aFdlH8M8EhrWRGxuzXZTgLGFadcAtAnWQljEvCB5LO/k7SujJ/p85JOT16PSGJdA7QANyfHrwPuSK5xHHBrm2t3LeMa1sk5keXL1mRpmNck/6A3tz0EfC4ipu70vmrOhawDjomIbbuIpWySTqCYFI+NiC2SHga67ebtkVx3/c7fgZn7yDqfqcBnkmV2kPRmST2BR4APJ31oQ4F37uKzjwOTJI1KPjsgOb4R6N3mfdOAz7XuSBqfvHwE+Ghy7D1A/xKx9gXWJUlsLMUaYas6oLVW+VGKTdYNwFJJ/5RcQ5KOKHEN2wc4kXU+V1Ps/5qj4sNTrqJY874TWJyc+zXF1T3eICJWAedQbMY9xetNu3uA01s7+4HPAxOTmwkLef3u6X9STIRPU2xivlQi1vuBBkmLgG9TTKStNlNckHABxT6wS5LjHwPOTuJ7Gji1jO/EOjmvfmFmuecamZnlnhOZmeWeE5mZ5Z4TmZnlnhOZmeWeE5mZ5Z4TmZnl3v8HPP2NPxQwir8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Results for highest accuracy model\n",
      "Trained DeepConvNet on single session using train/test split and got accuracy of: 0.9444444444444444\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEGCAYAAADmLRl+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbdElEQVR4nO3deZRU9Zn/8ffTGzQtNHQ3mywBI9Ggo2CI64SDSyI6EzU5TkZjHCfRH9FoNCY6P5LJxN8YzdHENUZNiLhkXIiCBhcEDYFRHEQWUVlEGFBkk7XZoburn98f97a22HRV0VV97y0+r3Puoe6tqu99us7x8ft97vd+r7k7IiJJVhR1ACIibaVEJiKJp0QmIomnRCYiiadEJiKJVxJ1AM3VVBX7gH6lUYcRW++93SnqEGLPivT/5tbsbtxBne+xtrRx5qkVvmlzKqPPzn177xR3H9mW82UiVolsQL9S3pjSL+owYuvMvl+KOoTYKyrvGHUIsfb6rufb3MbGzSlmTemb0WdLe/9vTZtPmIFYJTIRSQIn5Y1RB/EpSmQikhUHGonXRHolMhHJWiPqkYlIgjlOvYaWIpJkDqQ0tBSRpFONTEQSzYFUzFbNUSITkazFq0KmRCYiWXJcNTIRSTZ3qI9XHlMiE5FsGSnadLtmzimRiUhWHGhUj0xEkk49MhFJtGBCrBKZiCSYA/Uer3XflMhEJCuOkYrZ4tJKZCKStUbX0FJEEkw1MhEpAEZKNTIRSbJghVglMhFJMHejzoujDuNTlMhEJGuNqpGJSJIFxX4NLUUk0VTsF5GEU7FfRApCKmYTYuOVVkUk9hyj3ksy2tIxs2vNbKGZLTCzJ8yso5kNNLNZZrbMzP5sZmXp2lEiE5GsNBX7M9laY2Z9gKuBYe5+NFAMXADcCtzp7ocDW4BL08WkRCYiWXGMlGe2ZaAEKDezEqATsBY4DRgfvv8IcF4mjYiIZCWLYn+Nmc1ptj/G3ccAuPtqM7sNWAnsBl4C5gK17t4Qfn4V0CfdSZTIQs88UMOLj1XjDmddtJlv/p8NPPLrXsycUokZdK2p57q7VlLdqyF9YwXux7d9wAlnbKV2YwnfP2Nw1OHETmlZI795fAGlZY0UlzgzJlfz6G/7Rx1WzriTzfSLje4+rKU3zKwbcC4wEKgFngJGHkhMeRtamtmDZrbezBbk6xy58v67HXnxsWp++8J7/P6vS5j1chdWryjj/CvW8/upS7j/r0s44YxtPHpnr6hDjYWXnqri379zeNRhxFZ9nTH6X47iynOGcOU5x/Kl4bUcOWR71GHlTFDsL85oS+MMYIW7b3D3euBp4BSgazjUBOgLrE7XUD5rZA9zgNm1va1c2oEjh+6iYyenuASOOWkHr03qSkXnTx5Dumd3ERavK86RWTCrM9tr43WvXbwYe3YFv09JiVNS4sTswdxtlotiP8GQ8kQz62RmBpwOLAKmAeeHn7kEmJiuobwNLd39FTMbkK/2c2nAkXt4+NbebNtcTFnHRmb/rQuDjtkFwEO39OKvT1VR0SXFr8cvizhSSYqiIue3f3mLQ/vv4fnHerHkrc5Rh5QzjuVkYUV3n2Vm44F5QAPwJjAGeAEYZ2Y3hcfGpmtLNTKg/6C9fOsH6/nphZ+nY6dGDjtqN0Vhh+O7o9fx3dHrGHdPD559sDv/cv26aIOVRGhsNK46ZwgVnRv4j/ve5XODdvLB0oqow8qZXN1r6e43ADfsc3g5cHw27UQ+/cLMRpnZHDObs2FTKrI4Rn57M/dOeY/bn1nGIZUp+h6251Pvn/aNLcyYVBlRdJJUO7eX8PasSoYNr406lJwJnmtZlNHWXiJPZO4+xt2Hufuw7tXR1V1qNwad0/WrSnltUiWnfqOW1cs/mVA8c0ol/Q7fG1V4kiCVVfVUdA6ubpd1SDH05Fo+XF4ecVS5FDxpPJOtvWhoGbrxsgFs31JCcalz1a9WcUhlijt+0o9V/9uBoiLo0aeOq29dFXWYsTD6dys45qTtVFY18Ojsd/iv23szZVxN1GHFRrfudVz362UUFTlW5Lz6Yg1vTKuKOqycCR4HF6+LPXlLZGb2BDCCYELcKuAGd09btIvKHX/5bCH/Fw+83/6BJMAtVw2MOoRYe39JBVede2zUYeSNu7XrsDET+bxqeWG+2haRaGk9MhFJtGA9snhNqlQiE5EsaYVYEUm4YPqFemQikmBN91rGiRKZiGRNa/aLSKIFy/hoaCkiCacamYgkWrD6hYaWIpJgwS1KSmQikmjqkYlIAdDMfhFJNF21FJGCoKGliCRartbszyUlMhHJigMN6pGJSNJpaCkiyeYaWopIwmlhRREpCOqRiUiiaWFFEUk8x2hoVLFfRBJONTIRSTbX0FJEEk41MhEpCEpkIpJojpFSsV9Ekk7FfhFJNFexX0QKgSuRiUiy6aZxESkA6pG14r23O3HmoUOiDiO2xq7876hDiL1RQ8+JOoR429P2BOQOqcZ4JbJ4XUMVkURoxDLa0jGzrmY23szeNbPFZnaSmVWZ2ctmtjT8t1u6dpTIRCQrTjC0zGTLwN3AZHc/EjgWWAyMBqa6+yBgarjfKiUyEclSUOzPZGu1FbNKYDgwFsDd69y9FjgXeCT82CPAeekiUiITkay5Z7alMRDYADxkZm+a2QNmVgH0dPe14WfWAT3TNaREJiJZy2JoWWNmc5pto5o1UwIcB9zv7kOBnewzjHR3JxjNtipWVy1FJP6Cq5YZ94E2uvuw/by3Cljl7rPC/fEEiewjM+vt7mvNrDewPt1J1CMTkazlYmjp7uuAD83siPDQ6cAi4FngkvDYJcDEdPGoRyYiWcvhhNgfAo+ZWRmwHPguQQfrSTO7FPgA+Fa6RpTIRCQrTsZTK9K35T4faGnoeXo27SiRiUjW0l+QbF9KZCKSHQeP2S1KSmQikjXdNC4iiZfBZNd2td9EZmb30MpQ2N2vzktEIhJrTfdaxklrPbI57RaFiCSHA0lJZO7+SPN9M+vk7rvyH5KIxF3chpZpZ/aH6wMtAt4N9481s/vyHpmIxJThjZlt7SWTW5TuAs4ENgG4+1sES2+IyMHKM9zaSUZXLd39Q7NPZddUfsIRkdjzZBX7m3xoZicDbmalwDUEqziKyMEqaTUy4HLgSqAPsAYYEu6LyEHLMtzaR9oembtvBC5qh1hEJCkaow7g0zK5anmYmT1nZhvMbL2ZTTSzw9ojOBGJoaZ5ZJls7SSToeXjwJNAb+BQ4CngiXwGJSLxlqM1+3Mmk0TWyd3/y90bwu1RoGO+AxORGEvK9Aszqwpfvmhmo4FxBKH9MzCpHWITkbhK0PSLuQSJqyni7zd7z4Gf5isoEYk3i9n0i9butRzYnoGISEK4QRIXVjSzo4HBNKuNufuf8hWUiMRcUnpkTczsBmAEQSKbBJwFzACUyEQOVjFLZJlctTyf4Ikm69z9u8CxQGVeoxKReEvKVctmdrt7o5k1mFkXgqf+9stzXJEaNmIbl/9yDcVFzotPVPHk73pGHVIsvPTAobz6RE8w6HvkLr5323ssm9uFJ28eSEOdMeDvdvCvv1lKsRZQ56HJM9m9q5hUymhMGddcsL+HbSdQkhZWbGaOmXUF/khwJXMHMDPdl8ysH8HwsyfBnz7G3e8+8FDbR1GRc+WvVvPTCw5j49pS7pm0lNenVLJy6cE9dW7LujKmPnQov5w6j7KOjdx/xRG8PrEHE+/oz3VPvEOvw/bwl9v78z/je/KVCz6KOtxYGP29IWyrLYs6jLyI21XLtENLd/+Bu9e6+++BrwKXhEPMdBqAn7j7YOBE4EozG9y2cPPviKG7WPN+GetWdqChvojpE7ty0plbow4rFlINRt2eIlINULe7mA7lKUpKG+l12B4ABv99LXNfrI44SmkXMRta7jeRmdlx+25AFVASvm6Vu69193nh6+0ES//0yVXg+VLdq54Naz75v+jGtaXU9K6PMKJ46NarjjNHrebfTvwyPx52AuVdGvjy1zeSShnvv3UIAHMm1bB5TYeII40Hd7jpD29x959nM/L8NVGHk3PmmW3tpbWh5e2tvOfAaZmexMwGAEOBWS28NwoYBdCRTpk2Ke1sZ20x81+u4tbXZlPeJcX9VxzJ68905/u/W8K4GwfSUFfE4OFbKCqO2ZgjItdfchyb1negsqqOm8fMZ9WKTiyY2zXqsHInKTUydz81Fycws0OACcCP3H1bC+cZA4wB6GJVkf9XsGldKd0Prft4v6Z3PRvXlkYYUTwsmtGVmn576FzdAMCXRm5i2dwunPTNDYye8A4AC17pykfLy6MMMzY2rQ96pls3lzFzane+cPS2wklk7TxszEQm0y8OWLii7ATgMXd/Op/nypUl8zvRZ2AdPfvtpaS0kRHn1vL6S5ptUt1nL8vndWbv7iLcYfFrlRx6+C62bQySfP1e48X7+jLiO+sijjR6HcpTlHdq+Pj10JM388GyioijyrGY1cjydqHcgkX+xwKL3f2OfJ0n1xpTxr3/3odfPb6comJ4aVwVH7x3cF+xBDhs6A6+dPYmbjx7CEXFTv+jdjL82+t45rbP8fbUKhob4dTvrOOLp+jCSLfqOn5+V9BLLS52pk/qydzXCusiiMVsYUXzPC0aZGZ/D7wKvMMn60n+zN33u3JGF6vyE+z0vMRTCMaunBF1CLE3aug5UYcQazNrn2Zr/YY2Fbg69Ovnfa+5NqPPLr/+J3PdPe+T6DK5RckIlro+zN1vNLP+QC93f6O177n7DNpz0W4RaRftfUUyE5nUyO4DTgIuDPe3A/fmLSIRib+YLXWdSY3sBHc/zszeBHD3LWZWmNOVRSQzMeuRZZLI6s2smDB0M+tO7J6hIiLtKW5Dy0wS2W+BZ4AeZnYzwWoYP89rVCISXx6/q5aZPNfyMTObS7CUjwHnubueNC5yMEtajyy8SrkLeK75MXdfmc/ARCTGkpbIgBf45CEkHYGBwBLgqDzGJSIxlssaWViDnwOsdvd/NLOBBE9tqyZYOuxid69rrY1MlvH5O3c/Jvx3EHA8GaxHJiKSoWsIVsdpcitwp7sfDmwBLk3XQNb3WoZL85yQ7fdEpIDk6F5LM+sL/APwQLhvBCvrjA8/8ghwXrp2MqmR/bjZbhFwHFB4CyyJSGayu2pZY2Zzmu2PCVe8aXIX8G9A53C/Gqh194ZwfxUZrGOYSY2sc7PXDQQ1swkZfE9EClXmNbKN+7vX0sz+EVjv7nPNbERbwmk1kYVFuM7ufl1bTiIihcPIWbH/FOAcMzub4EJiF+BuoKuZlYS9sr7A6nQNtbbUdYm7p8KTiYh8Igc1Mnf/qbv3dfcBwAXA39z9ImAawcR7gEuAienCaa1H9gZBPWy+mT0LPAXsbBZEIhZKFJEcy//qF/8XGGdmNwFvEqxr2KpMamQdgU0EVxKa5pM5oEQmcrDK8S1K7j4dmB6+Xk4wzStjrSWyHuEVywV8ksA+Pm9WUYpIQUnSTePFwCG0vDhizP4MEWlXMcsArSWyte5+Y7tFIiLJEMOnKLWWyLRMtYi0KElDSz0FRERalpRE5u6b2zMQEUmOxC2sKCLyKQmrkYmIfIYRvwK6EpmIZE89MhFJuiRdtRQRaZkSmYgkWhIfByci8hnqkYlI0qlGJiLJp0S2f1ZSTHHXqqjDiK3LBumusXQeXfZc+g8dxL529tactKMemYgkm5PzhRXbSolMRLKSw4eP5IwSmYhkT4lMRJLOPF6ZTIlMRLKj1S9EpBCoRiYiiadblEQk+dQjE5FEy/+TxrOmRCYi2VMiE5Ek04RYESkI1hivTKZEJiLZ0TwyESkEmn4hIsmnHpmIJJ2K/SKSbA7opnERSTrVyEQk0TSPTESSz11DSxFJPvXIRCT5YpbIiqIOQESSxzyzrdU2zPqZ2TQzW2RmC83smvB4lZm9bGZLw3+7pYtHiUxEsuNAyjPbWtcA/MTdBwMnAlea2WBgNDDV3QcBU8P9VimRiUjWctEjc/e17j4vfL0dWAz0Ac4FHgk/9ghwXrp4VCMTkexlftWyxszmNNsf4+5j9v2QmQ0AhgKzgJ7uvjZ8ax3QM91JlMhEJGtZXLXc6O7DWm3L7BBgAvAjd99mZh+/5+5ulv5sGlqKSHY8iy0NMyslSGKPufvT4eGPzKx3+H5vYH26dpTIRCQrBljKM9pabSfoeo0FFrv7Hc3eeha4JHx9CTAxXUwaWopI1nL0pPFTgIuBd8xsfnjsZ8AtwJNmdinwAfCtdA0pkYlIdnK0Qqy7zyDo4LXk9GzaUiJrwUOTZ7J7VzGplNGYMq65oNVa5UGnpvderr99OV1r6sGNSU90Z+LDvaIOK3KTx/Zm2uM9cYxTL1zHWZetZceWEu658gg2fNiB7v32cvV971LRNRV1qG10EN1raWYdgVeADuF5xrv7Dfk6X66N/t4QttWWRR1GLDU2GH+8uT/LFlZQXpHinucW8OaMSlYuK486tMh8+G4npj3ekxuff5uS0kZuvfgohp6+hb893pOjTqnlnCtX8+y9fXj2vr5c+LMPog63zeJ2r2U+i/17gdPc/VhgCDDSzE7M4/mknWzeUMayhRUA7N5ZzIfLyqnuVRdxVNFas6yczw/dQYfyRopL4IsnbGX25GrmvVTNV84PLrp95fz1zJ1SHXGkOdK0Aka6rZ3kLZF5YEe4WxpuMcvjLXOHm/7wFnf/eTYjz18TdTix1rPPXj4/eBdL5h8SdSiR6nvELpa80YXtW0rYu7uI+dO6sXlNGVs3ltKtZz0AXXvUs3VjacSR5oDn5qplLuW1RmZmxcBc4HDgXneflc/z5cr1lxzHpvUdqKyq4+Yx81m1ohML5naNOqzY6dgpxc/vX8offtmfXTuKow4nUn0G7ebrP1jFLRcdRYfyFJ8bvJOifX4SM/Zf2k6amHVJ8jqPzN1T7j4E6Ascb2ZH7/sZMxtlZnPMbE5d4558hpOxTes7ALB1cxkzp3bnC0dvizii+CkuaeQ/7l/KtInVvDalKupwYmHEBeu5edJb/GLCAioqG+g1cDeVNfVs+SjohW35qJTK6vqIo8wNc89oay/tMiHW3WuBacDIFt4b4+7D3H1YWVHH9ginVR3KU5R3avj49dCTN/PBsoqIo4ob59pbV7ByWTlPj+0ddTCx0TRs3Li6jNmTqzn5vA0c99XNvDq+BwCvju/BcV/bFGWIuROzGlk+r1p2B+rdvdbMyoGvArfm63y50q26jp/f9Q4AxcXO9Ek9mftagRRoc+SoYTs445ubWPFuOfe+sACAh3/Tl9nTu0YbWMTuHnUE22tLKSlx/vWm5VRUpvj6lau454ojmD6uJzV993L1fUuiDrPtHDiIHj7SG3gkrJMVAU+6+/N5PF9OrFtVzlXnHx91GLG2cE5nRg7Ub7SvXzy94DPHOndr4GfjFkYQTf4Y7TtszETeEpm7v02wLIeIFJrGeHXJNLNfRLJzkA0tRaRAHTRDSxEpYEpkIpJsB9FN4yJSoJqeohQjSmQikjXVyEQk+ZTIRCTRHGhUIhORRFOxX0QKgRKZiCSaA6l4Te1XIhORLDm4EpmIJJ2GliKSaLpqKSIFQT0yEUk8JTIRSTR3SMXraelKZCKSPfXIRCTxlMhEJNlcVy1FJOEcXBNiRSTxdIuSiCSaux4HJyIFQMV+EUk6V49MRJJNCyuKSNLppnERSToHPGa3KBVFHYCIJIyHCytmsqVhZiPNbImZLTOz0QcaknpkIpI1z8HQ0syKgXuBrwKrgNlm9qy7L8q2LfXIRCR7uemRHQ8sc/fl7l4HjAPOPZBwzGN09cHMNgAfRB1HMzXAxqiDiDH9PunF7Tf6nLt3b0sDZjaZ4O/KREdgT7P9Me4+JmznfGCku18W7l8MnODuV2UbU6yGlm39gXPNzOa4+7Co44gr/T7pFeJv5O4jo45hXxpaikhUVgP9mu33DY9lTYlMRKIyGxhkZgPNrAy4AHj2QBqK1dAyhsZEHUDM6fdJT7/Rfrh7g5ldBUwBioEH3X3hgbQVq2K/iMiB0NBSRBJPiUxEEk+JrAVm9qCZrTezBVHHEkdm1s/MppnZIjNbaGbXRB1TnJhZRzN7w8zeCn+f/4w6pkKnGlkLzGw4sAP4k7sfHXU8cWNmvYHe7j7PzDoDc4HzDuTWkkJkZgZUuPsOMysFZgDXuPvrEYdWsNQja4G7vwJsjjqOuHL3te4+L3y9HVgM9Ik2qvjwwI5wtzTc1GPIIyUyaRMzGwAMBWZFHEqsmFmxmc0H1gMvu7t+nzxSIpMDZmaHABOAH7n7tqjjiRN3T7n7EILZ6sebmUoUeaREJgckrP1MAB5z96ejjieu3L0WmAbE7v7EQqJEJlkLi9ljgcXufkfU8cSNmXU3s67h63KC9bbejTSoAqdE1gIzewKYCRxhZqvM7NKoY4qZU4CLgdPMbH64nR11UDHSG5hmZm8T3E/4srs/H3FMBU3TL0Qk8dQjE5HEUyITkcRTIhORxFMiE5HEUyITkcRTIksQM0uFUx0WmNlTZtapDW09HD7FBjN7wMwGt/LZEWZ28gGc430z+8zTdvZ3fJ/P7Gjt/RY+///M7LpsY5TCoESWLLvdfUi4IkcdcHnzN83sgJYud/fL0qxcMQLIOpGJtBclsuR6FTg87C29ambPAovCm5V/Y2azzextM/s+BLPxzex34ePp/wr0aGrIzKab2bDw9UgzmxeupTU1vCn8cuDasDf4lXDm+oTwHLPN7JTwu9Vm9lK4BtcDgKX7I8zsL2Y2N/zOqH3euzM8PtXMuofHPm9mk8PvvGpmR+bk15RE08NHEijseZ0FTA4PHQcc7e4rwmSw1d2/bGYdgNfM7CWCFSqOAAYDPYFFwIP7tNsd+CMwPGyryt03m9nvgR3uflv4uceBO919hpn1J3h4xBeBG4AZ7n6jmf0DkMkdEd8Lz1EOzDazCe6+CagA5rj7tWb2i7Dtqwge5nG5uy81sxOA+4DTDuBnlAKiRJYs5eHSMBD0yMYSDPnecPcV4fGvAcc01b+ASmAQMBx4wt1TwBoz+1sL7Z8IvNLUlrvvb022M4DBwS2XAHQJV8IYDnwz/O4LZrYlg7/pajP7Rvi6XxjrJqAR+HN4/FHg6fAcJwNPNTt3hwzOIQVOiSxZdodLw3ws/A96Z/NDwA/dfco+n8vlvZBFwInuvqeFWDJmZiMIkuJJ7r7LzKYDHffzcQ/PW7vvbyCiGlnhmQJcES6zg5l9wcwqgFeAfw5raL2BU1v47uvAcDMbGH63Kjy+Hejc7HMvAT9s2jGzIeHLV4Bvh8fOArqlibUS2BImsSMJeoRNioCmXuW3CYas24AVZvZP4TnMzI5Ncw45CCiRFZ4HCOpf8yx4eMofCHrezwBLw/f+RLC6x6e4+wZgFMEw7i0+Gdo9B3yjqdgPXA0MCy8mLOKTq6f/SZAIFxIMMVemiXUyUGJmi4FbCBJpk50ECxIuIKiB3Rgevwi4NIxvIXBuBr+JFDitfiEiiacemYgknhKZiCSeEpmIJJ4SmYgknhKZiCSeEpmIJJ4SmYgk3v8HoMG9YFpyVmMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################################\n",
    "# SINGLE TRIAL SHALLOWCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Configure\n",
    "retrain_model = False\n",
    "tensorboard_name = \"DeepCN_singlesession_C0_100hz_06drop\"\n",
    "best_model_filename = f\"./saved_variables/6/DeepConvNet/{tensorboard_name}\"\n",
    "\n",
    "# Get train/test split of data from one session\n",
    "X_train, X_test, y_train, y_test = train_test_split(mne_fixed_window_epochs_data, ohe_labels, \n",
    "                                                    test_size = 0.3,\n",
    "                                                    shuffle= True,\n",
    "                                                    stratify= ohe_labels,                                                    \n",
    "                                                    random_state=98)\n",
    "\n",
    "\n",
    "# Train on train/test split of data from one session\n",
    "## Note: the model is forced to use GPU, if GPU is not available replace with what is available e.g. /cpu:0\n",
    "if (retrain_model): # Retrain or not\n",
    "    with tf.device('/gpu:0'):\n",
    "        keras_deepcn_model.fit(\n",
    "            x= X_train,\n",
    "            y= y_train,\n",
    "            batch_size= 128, # Default: 32\n",
    "            epochs= 500, # Default: 500 (EEGNet paper)\n",
    "            verbose= 1, # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "            #callbacks= [tensorboard_callback(\"EEGNet_raw_signal_0.5s_100samps_50kernlen_02nr\")], # To be used for TF Board\n",
    "            callbacks= [tensorboard_callback(tensorboard_name),\n",
    "                        lowest_loss_model_save_callback(best_model_filename),\n",
    "                        highest_accuracy_model_save_callback(best_model_filename)],\n",
    "            validation_split= 0.3,\n",
    "            shuffle= True,\n",
    "            sample_weight= None, # Can be interesting due to time series\n",
    "            use_multiprocessing=True, # Done for faster speed\n",
    "            workers= 4 # Done for faster speed\n",
    "            )\n",
    "\n",
    "# Convert labels back to original\n",
    "y_test = ohe.inverse_transform(y_test)\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"Results for lowest loss model\")\n",
    "keras_deepcn_model = load_lowest_loss_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_deepcn_model.predict(X_test)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained DeepConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"\\n\\nResults for highest accuracy model\")\n",
    "keras_deepcn_model = load_highest_accuracy_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_deepcn_model.predict(X_test)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained DeepConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Remove unused variables\n",
    "del X_train\n",
    "del X_test\n",
    "del y_train\n",
    "del y_test\n",
    "del accuracy\n",
    "del y_pred\n",
    "del retrain_model\n",
    "del tensorboard_name\n",
    "del best_model_filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "86df9113",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAUP\n",
    "####################################################\n",
    "\n",
    "# delete unused variables\n",
    "del keras_deepcn_model\n",
    "del mne_fixed_window_epochs_data\n",
    "del ohe\n",
    "del ohe_labels\n",
    "del mne_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c186b1f",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Fixed window classification: Multi trial training with unseen trial testing | DeepConvNet | three class MI task | 100Hz input signal\n",
    "\n",
    "Being outperformed by both ShallowConvNet and EEGNet, it is studied if DeepConvNet can proof to be better in inter-session performance then any of the other two studies DL approaches thus far.\n",
    "\n",
    "**Remember the results for subject `C`**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** | **DeepConvNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|-----------------------------------|--------------------------------|\n",
    "| 0              | 1 + 2           | 0.782                        | 0.916 / 0.920             | 0.903 / 0.910                     | 0.895 / 0.909                  |\n",
    "| 1              | 0 + 2           | 0.635                        | 0.852 / 0.860             | 0.782 / 0.787                     | 0.822 / 0.831                  |\n",
    "| 2              | 1 + 2           | 0.347                        | 0.607 / 0.594             | 0.509 / 0.489                     | 0.625 / 0.596                  |\n",
    "| 0              | 1               | 0.667                        | 0.891 / 0.860             | 0.872 / 0.872                     | 0.863 / 0.903                  |\n",
    "| 1              | 0               | 0.662                        | 0.794 / 0.783             | 0.713 / 0.731                     | 0.742 / 0.755                  |\n",
    "\n",
    "**For subject `B` these results are**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** | **DeepConvNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|-----------------------------------|--------------------------------|\n",
    "| 0              | 1 + 2           | 0.409                        | 0.498 / 0.489             | 0.447 / 0.448                     | 0.461 / 0.458                  |\n",
    "| 1              | 0 + 2           | 0.497                        | 0.640 / 0.654             | 0.570 / 0.571                     | 0.679 / 0.671                  |\n",
    "| 2              | 1 + 2           | 0.502                        | 0.677 / 0.660             | 0.602 / 0.601                     | 0.659 / 0.659                  |\n",
    "\n",
    "**For subject `E` the results are**\n",
    "\n",
    "| **Test index** | **Train index** | **CSP + SVM: test accuracy** | **EEGNet: test accuracy** | **ShallowConvNet: test accuracy** | **DeepConvNet: test accuracy** |\n",
    "|----------------|-----------------|------------------------------|---------------------------|-----------------------------------|--------------------------------|\n",
    "| 0              | 1 + 2           | 0.398                        | 0.714 / 0.713             | 0.656 / 0.681                     | 0.695 / 0.682                  |\n",
    "| 1              | 0 + 2           | 0.371                        | 0.667 / 0.646             | 0.641 / 0.713                     | 0.533 / 0.571                  |\n",
    "| 2              | 1 + 2           | 0.475                        | 0.734 / 0.688             | 0.699 / 0.712                     | 0.655 / 0.759                  |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "64613239",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading 0 ... 666799  =      0.000 ...  3333.995 secs...\n",
      "Reading 0 ... 681199  =      0.000 ...  3405.995 secs...\n",
      "Reading 0 ... 669399  =      0.000 ...  3346.995 secs...\n",
      "Loaded test epochs and extracted labels\n",
      "Loaded train epochs and extracted labels\n",
      "Loaded train epochs and extracted labels\n",
      "\n",
      "\n",
      "Concatenated the train epochs\n",
      "Total amount of train labels: 1919\n",
      "\n",
      "\n",
      "Got the test epochs\n",
      "Total amount of train labels: 960\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Lennert\\AppData\\Local\\Temp\\ipykernel_16324\\2395505687.py:29: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.\n",
      "  mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
      "C:\\Users\\Lennert\\AppData\\Local\\Temp\\ipykernel_16324\\2395505687.py:34: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.\n",
      "  mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OHE test and train labels, train labels:\n",
      "[[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [1. 0. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 1. 0.]]\n",
      "Train labels OHE match regular labels: True\n",
      "Test labels OHE match regular labels: True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# PREPPING THE DATA\n",
    "####################################################\n",
    "\n",
    "# Get data and choose test trial\n",
    "mne_raws = load_mne_raws(subject= \"C\")\n",
    "test_trial = mne_raws[0]\n",
    "\n",
    "# Init variables\n",
    "epochs = []\n",
    "labels_train = []\n",
    "\n",
    "for single_mne_raw in mne_raws:\n",
    "    # Load epochs from raw\n",
    "    mne_fixed_window_epochs = CLA_dataset.get_usefull_epochs_from_raw(single_mne_raw, start_offset=-1.5, end_offset=1.5)['task/neutral', 'task/left', 'task/right']\n",
    "    with io.capture_output():\n",
    "        mne_fixed_window_epochs.load_data()\n",
    "    \n",
    "    if (single_mne_raw == test_trial):\n",
    "        mne_fixed_window_epochs_test = mne_fixed_window_epochs\n",
    "        labels_test = mne_fixed_window_epochs.events[:, -1]\n",
    "        print(\"Loaded test epochs and extracted labels\")\n",
    "    else:  \n",
    "        epochs.append(mne_fixed_window_epochs)\n",
    "        labels_train.extend(mne_fixed_window_epochs.events[:, -1])\n",
    "        print(\"Loaded train epochs and extracted labels\")\n",
    "    \n",
    "# Make single epoch object for training\n",
    "mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
    "print(\"\\n\\nConcatenated the train epochs\")\n",
    "print(f\"Total amount of train labels: {len(labels_train)}\")\n",
    "    \n",
    "# Show test epoch object\n",
    "mne_fixed_window_epochs_train = mne.concatenate_epochs(epochs, verbose=False)\n",
    "print(\"\\n\\nGot the test epochs\")\n",
    "print(f\"Total amount of train labels: {len(labels_test)}\")\n",
    "\n",
    "# Go to 2D representation\n",
    "labels_train = np.array(labels_train).reshape(-1, 1)\n",
    "labels_test = np.array(labels_test).reshape(-1, 1)\n",
    "\n",
    "# One Hot Encode the labels\n",
    "ohe = OneHotEncoder()\n",
    "ohe_labels_train = ohe.fit_transform(labels_train).toarray()\n",
    "ohe_labels_test = ohe.transform(labels_test).toarray()\n",
    "\n",
    "\n",
    "# Show ohe labels\n",
    "print(f\"OHE test and train labels, train labels:\\n{ohe_labels_train[:10]}\")\n",
    "\n",
    "# Validate OHE\n",
    "print(f\"Train labels OHE match regular labels: {np.array_equal(ohe.inverse_transform(ohe_labels_train), labels_train)}\")\n",
    "print(f\"Test labels OHE match regular labels: {np.array_equal(ohe.inverse_transform(ohe_labels_test), labels_test)}\")\n",
    "\n",
    "# Get training data\n",
    "mne_fixed_window_epochs_train_data = mne_fixed_window_epochs_train.get_data(tmin=0.2, tmax=0.7)\n",
    "mne_fixed_window_epochs_test_data = mne_fixed_window_epochs_test.get_data(tmin=0.2, tmax=0.7)\n",
    "\n",
    "# Fix scaling sensitivity as MNE stores as data * 10e-6\n",
    "mne_fixed_window_epochs_train_data = mne_fixed_window_epochs_train_data * 1000000\n",
    "mne_fixed_window_epochs_test_data = mne_fixed_window_epochs_test_data * 1000000\n",
    "\n",
    "# Remove unused variables\n",
    "del epochs\n",
    "del mne_fixed_window_epochs\n",
    "del test_trial\n",
    "del single_mne_raw\n",
    "del mne_raws\n",
    "del mne_fixed_window_epochs_train\n",
    "del mne_fixed_window_epochs_test\n",
    "del labels_train\n",
    "del labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1420f3dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_5\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_6 (InputLayer)        [(None, 21, 100, 1)]      0         \n",
      "                                                                 \n",
      " conv2d_11 (Conv2D)          (None, 21, 97, 25)        125       \n",
      "                                                                 \n",
      " conv2d_12 (Conv2D)          (None, 1, 97, 25)         13150     \n",
      "                                                                 \n",
      " batch_normalization_12 (Bat  (None, 1, 97, 25)        100       \n",
      " chNormalization)                                                \n",
      "                                                                 \n",
      " activation_15 (Activation)  (None, 1, 97, 25)         0         \n",
      "                                                                 \n",
      " max_pooling2d_4 (MaxPooling  (None, 1, 48, 25)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_6 (Dropout)         (None, 1, 48, 25)         0         \n",
      "                                                                 \n",
      " conv2d_13 (Conv2D)          (None, 1, 44, 50)         6300      \n",
      "                                                                 \n",
      " batch_normalization_13 (Bat  (None, 1, 44, 50)        200       \n",
      " chNormalization)                                                \n",
      "                                                                 \n",
      " activation_16 (Activation)  (None, 1, 44, 50)         0         \n",
      "                                                                 \n",
      " max_pooling2d_5 (MaxPooling  (None, 1, 22, 50)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_7 (Dropout)         (None, 1, 22, 50)         0         \n",
      "                                                                 \n",
      " conv2d_14 (Conv2D)          (None, 1, 18, 100)        25100     \n",
      "                                                                 \n",
      " batch_normalization_14 (Bat  (None, 1, 18, 100)       400       \n",
      " chNormalization)                                                \n",
      "                                                                 \n",
      " activation_17 (Activation)  (None, 1, 18, 100)        0         \n",
      "                                                                 \n",
      " max_pooling2d_6 (MaxPooling  (None, 1, 9, 100)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_8 (Dropout)         (None, 1, 9, 100)         0         \n",
      "                                                                 \n",
      " conv2d_15 (Conv2D)          (None, 1, 5, 200)         100200    \n",
      "                                                                 \n",
      " batch_normalization_15 (Bat  (None, 1, 5, 200)        800       \n",
      " chNormalization)                                                \n",
      "                                                                 \n",
      " activation_18 (Activation)  (None, 1, 5, 200)         0         \n",
      "                                                                 \n",
      " max_pooling2d_7 (MaxPooling  (None, 1, 2, 200)        0         \n",
      " 2D)                                                             \n",
      "                                                                 \n",
      " dropout_9 (Dropout)         (None, 1, 2, 200)         0         \n",
      "                                                                 \n",
      " flatten_3 (Flatten)         (None, 400)               0         \n",
      "                                                                 \n",
      " dense_3 (Dense)             (None, 3)                 1203      \n",
      "                                                                 \n",
      " activation_19 (Activation)  (None, 3)                 0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 147,578\n",
      "Trainable params: 146,828\n",
      "Non-trainable params: 750\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE DEEPCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Create the TensorFlow Keras model\n",
    "keras_deepcn_model = DeepConvNet(\n",
    "    nb_classes = 3, # int, number of classes to classify. \n",
    "    Chans = 21, # number of channels in the EEG data. \n",
    "    Samples = 100, # number of time points in the EEG data. (default: 2s 250, paper: 1s 250)\n",
    "    dropoutRate = 0.6, # dropout fraction. (default: 0.5)\n",
    "    conv_filters = 4, # Conv2D kernel size (default: 5, paper: 10)\n",
    "    strides = 2, # Stride size for average pooling layer (default: 2, paper: 3)\n",
    "    pool_size = 2 # Pool size for average pooling layer (default: 2, paper: 3)\n",
    "    )\n",
    "\n",
    "# Compile the model so it can be fitted\n",
    "# Loss and optimizer from EEGNet paper\n",
    "keras_deepcn_model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics=[\"accuracy\"])\n",
    "\n",
    "# Show summary of the model\n",
    "keras_deepcn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "adef1fca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for lowest loss model\n",
      "Trained DeepConvNet on single session using train/test split and got accuracy of: 0.8947916666666667\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAg40lEQVR4nO3deZwdZZn28d/Vnc6+kZVsEJYAJkACBgwww7AMEkQEHBQYR1CZN4iIMIMLMCKK44Ys4jDihEVAkW3YEYMBYQBlS0JYkhAIELInZN+TXu73j1MNbUi6z+nu03VO9fXlU5/UeapO1d1F+s6zVD2liMDMLIsq0g7AzKxYnODMLLOc4Mwss5zgzCyznODMLLM6pB1AQ737VMagoSUVUklZ+Fq3tEMoeercKe0QStqm6jVsrdmolhzj2CO7xYqVtXntO/XVLY9FxPiWnK8lSiqbDBragd88PCjtMErWpbsdlHYIJa9y9xFph1DSnnvnNy0+xvKVtbzw2NC89q0a9Ha/Fp+wBUoqwZlZOQhqoy7tIPLiPjgzK0gAdUReS2MkdZb0oqRXJM2Q9IOkfDdJL0iaI+kuSR2T8k7J5znJ9uFNxeoEZ2YFq8vzvyZsAY6KiNHAGGC8pHHAz4BrImJPYBVwVrL/WcCqpPyaZL9GOcGZWUGCoDrq8loaPU7O+uRjVbIEcBTwv0n5rcBJyfqJyWeS7UdLanTAxAnOzAoSQC2R1wL0kzSlwTKh4bEkVUqaDiwDJgNvA6sjoibZZQEwJFkfAswHSLavAfo2FqsHGcysYE31rzWwPCLG7mhjRNQCYyT1Bu4H9ml5dB9ygjOzggRQ28qzEEXEaklPAocAvSV1SGppQ4GFyW4LgWHAAkkdgF7AisaO6yaqmRWsLs+lMZL6JzU3JHUBjgFmAU8CpyS7nQk8mKw/lHwm2f7naGK+N9fgzKwg8WH/WksNAm6VVEmusnV3RDwiaSZwp6T/BF4Gbkr2vwn4raQ5wErgtKZO4ARnZgWJgOpWyG8R8SpwwHbK3wEO3k75ZuBzhZzDCc7MCiRqadHjrG3GCc7MChJAXZm86cAJzswK5hqcmWVS7kZfJzgzy6AAqqM87jBzgjOzggSitkxuoXWCM7OC1YWbqGaWQe6DM7MME7XugzOzLMrN6OsEZ2YZFCG2RmXaYeTFCc7MClbnPjgzy6LcIIObqGaWSR5kMLOM8iCDmWVarW/0NbMsCkR1lEfqKI8ozaxkeJDBzDIrkJuoZpZdHmQocWsWdeTeC3dj/fIqEBx0+vsc8uWlLJ7ZhYf+Yzg1Wyqo6BCccPl7DB2zgVce6MMzvx5EAJ261XHCD+cyaOSmtH+MVPQfvJVvXTuP3v1rIODR3/XlgZv6px1WqqqqarniF09RVVVHZWXw7NNDuP3WUYw+YClnnf0aUrB5UweuvuIgFi/qnna4LRKBbxORdDPwaWBZROxbrPM0V0WHYPx/zGfwvhvZsr6C608YxR5/t4bHfjKMI89fxF5HrOHNJ3vx2E+Hctads9lp2FbOuusNuvSq5c2nevHQJcM5+4FZaf8YqaitERMvH8yc17rSpVst1016k2lP92DeW53TDi011dUVXHzhP7B5cwcqK+u48tonmfLiznz9gpe5/NJDmT+vJ8d/5m1O+5dZXHPFQWmH2yK5QYbyeFSrmGn4FmB8EY/fIj0GVDN4340AdOpeR/89N7F2SUck2LI+9z9v87pKegysBmCXj6+nS69aAIYdsJ41SzqmE3gJWLmsijmvdQVg04ZK5s/pTL9B1SlHlTaxeXOuvtChQx2VHQIiV9vp2jV3bbp1q2blimz8I1BLRV5L2opWg4uIpyUNL9bxW9OqBR1ZPLMrQ8es57jvzeO2M/di0o+HEXUw4X8/Wkubeld/9vqHNSlEWnoGDt3KHvtu4o1pXdMOJXUVFcG11z/O4CHreeTBPZj9Rl+uverj/OAnf2Hrlko2buzAv339qLTDbLFAZTPhZfopNmVbNlRw5zl7ctyl8+nco46XfjeA4747n2/99RWO++487r9o+N/s/85zPZh6dz8+edH8dAIuIZ271nLpjXP59fcGs3F9eTRZiqmuTpx39jGccerx7LXPKnYdvoaT/uktLrv4MM447XgmTxrOhHNeSTvMVlEuNbjUI5A0QdIUSVNWr6xt03PXVos7z9mT/U9cwajxqwB4+b6+jEzW9z1+FQtf+bBDeMmsLjxw0XC+MPEtuu7UtrGWmsoOwaU3zuXP9+3EX/7YO+1wSsqGDR15dXp/xh68hN33WMPsN/oC8PRTw/jYqBUpR9dyufeiVuS1pC31CCJiYkSMjYixvfu0XS0gAu7/znD677mJw/516QflPQZUM/eFHgC889ce9B2+GYDVCztyxzl7csrV79Jv9y1tFmdpCv79qvnMf6sz901s36On9Xr22kK3blsB6NixlgM+vpT583rQtVs1Q4auA8iVvdczzTBbSe7N9vksjR5FGibpSUkzJc2QdH5S/n1JCyVNT5ZPNfjOxZLmSJot6dimIm23t4nMm9KdV+7vx8C9N/LfnxoFwDHfWsBJP5nLo5fvQm2NqOpUx2d+PBeAp345mI2rOvDwpbsCuVHYcx6amVb4qRp18Ab+8XOreGdmZ341eTYAv/nJIF76cxZ+eZunT99NXPjtKVRUBlLwzP8N5cXnB/PLqz7Of1z2HHUh1q+r4hdXjk071BbLvTawVSojNcCFETFNUg9gqqTJybZrIuLKhjtLGgmcBowCBgOPS9orInbYnFJEtEagHz2wdAdwBNAPWApcFhE3Nfadj+3fKX7z8KCixJMFl+5W3rcXtIXKj41IO4SS9tw7v2HNpsUtGiEYMqp3fO3uv8tr3+/u+4epEZFXVpf0IHAdcBiwfjsJ7mKAiPhJ8vkx4PsR8dyOjlnMUdTTi3VsM0tXa9/om9xxcQDwArkE93VJZwBTyNXyVgFDgOcbfG1BUrZDqffBmVl5yc0Hp7wWoF/9IGKyTNj2eJK6A/cCF0TEWuB6YA9gDLAYuKq5sbbbPjgza66CZvRd3lgTVVIVueR2e0TcBxARSxtsvwF4JPm4EBjW4OtDk7Idcg3OzAqSu01EeS2NkSTgJmBWRFzdoLxhR/zJwOvJ+kPAaZI6SdoNGAG82Ng5XIMzs4K04rOohwFfBF6TND0puwQ4XdIYcrl0LnA2QETMkHQ3MJPcCOy5jY2gghOcmTVDa0yXFBHPwnZvlnu0ke/8CPhRvudwgjOzguSmSyqPZ1Gd4MysYOXysL0TnJkVJDebSHmMTzrBmVlBco9qOcGZWSa5BmdmGVbXxEwhpcIJzswK4lFUM8s0N1HNLJPK6Z0MTnBmVpAAalyDM7OschPVzLIpj5lCSoUTnJkVpH7Cy3LgBGdmBXMNzswyqX7Cy3LgBGdmBQlETZ0HGcwso9wHZ2bZFG6imllGuQ/OzDLNCc7MMikQtR5kMLOs8iCDmWVSeJDBzLIsnODMLJv8sL2ZZZhrcM2w8LVuXLrbQWmHUbImvPlO2iGUvBs/u0/aIZQ2tTwxRUBtXXkkuPIY6zWzklKH8loaI2mYpCclzZQ0Q9L5SXkfSZMlvZX8uVNSLkm/lDRH0quSDmwqTic4MytIkGui5rM0oQa4MCJGAuOAcyWNBC4CnoiIEcATyWeA44ARyTIBuL6pEzjBmVmBcoMM+SyNiYjFETEtWV8HzAKGACcCtya73QqclKyfCNwWOc8DvSUNauwcJdUHZ2blISLvXftJmtLg88SImLjtTpKGAwcALwADI2JxsmkJMDBZHwLMb/C1BUnZYnbACc7MClbAKOryiBjb2A6SugP3AhdExFo1GAiJiJCUfzrdhhOcmRUkN4raOr1bkqrIJbfbI+K+pHippEERsThpgi5LyhcCwxp8fWhStkPugzOzgkXktzRGuaraTcCsiLi6waaHgDOT9TOBBxuUn5GMpo4D1jRoym6Xa3BmVrBWutH3MOCLwGuSpidllwA/Be6WdBbwHvD5ZNujwKeAOcBG4MtNncAJzswKEuR1C0jTx4l4FnZ4s9zR29k/gHMLOYcTnJkVrNm9/m3MCc7MChMQZfKolhOcmRXMD9ubWWYVcKNvqnaY4CT9F400tSPiG0WJyMxKWv2zqOWgsRrclEa2mVl7FUC5J7iIuLXhZ0ldI2Jj8UMys1JXLk3UJp9kkHSIpJnAG8nn0ZJ+VfTIzKxEiajLb0lbPo9q/QI4FlgBEBGvAIcXMSYzK3WR55KyvEZRI2K+/naq49rihGNmJS+yMchQb76kQ4FInvw/n9zEdGbWXpVA7Swf+TRRv0ru+a8hwCJgDAU+D2ZmWaM8l3Q1WYOLiOXAF9ogFjMrF3VpB5CffEZRd5f0sKT3JS2T9KCk3dsiODMrQfX3weWzpCyfJurvgbuBQcBg4B7gjmIGZWalrTUmvGwL+SS4rhHx24ioSZbfAZ2LHZiZlbByv01EUp9k9Y+SLgLuJBfyqeRm1jSz9qoEmp/5aGyQYSq5hFb/k5zdYFsAFxcrKDMrbc1/z1XbauxZ1N3aMhAzKxMhKIHHsPKR15MMkvYFRtKg7y0ibitWUGZW4sq9BldP0mXAEeQS3KPAccCzgBOcWXtVJgkun1HUU8i94WZJRHwZGA30KmpUZlbayn0UtYFNEVEnqUZST3JvmR7W1JfKVVWnOq66bw5VHYPKDsEzf+jNb6/cOe2w2tz6xZU8+e0BbFpeiQT7nLqW/c5cC8Drt/Vkxu09qaiEYUdsZNy3V1K7FZ75Xn/ef70TUnDod1cw+BObU/4p2l5FRR2/vG4yy5d34fvfO5xvX/QcI0asoqZWvPlGX3557Vhqa8v8fetZmPCygSmSegM3kBtZXQ8819SXJA0j14wdSO6STIyIa5sfatuo3iK+/bk92LyxksoOwdUPzOGlP/fgjWnd0g6tTVVUwiEXraDfqK1sXS/u/+wQhh62iU3LK3nvia6c8vACKjvCphW5X9Y37u4JwOceWcCmFRX88V8HcfK9C1GZ/y4X6sST32LevJ507VoNwJNP7MoVPx0HwHcufp7xx73DHx7ZM80QW0W5jKI2+dcvIr4WEasj4tfAMcCZSVO1KTXAhRExEhgHnCtpZMvCbQti88ZKADpUBZVVURJ3ZLe1rgNq6TdqKwAduwe996hmw9IOzLyjJ6MnrKGyY26/Ln1zDyWumlPF4HGbPijr2KOO91/rlErsaenXbyMHH7yIxyZ9+CTjSy8Npv7B89mz+9CvX0YmxS6TJuoOE5ykA7ddgD5Ah2S9URGxOCKmJevryE2xNKS1Ai+miorgV5Nnc9erM3j56e7Mfrl91d62tW5BB5bP7MSA0ZtZ824VS6Z05v5TBvPwFwax7NVcEuu7z1be+3NX6mpg7fwOLJ/RkfVL2tdL284+52VuunE0ddu5haKyso6jj57LlCmDUois9SnyW9LW2N/AqxrZFsBR+Z5E0nDgAOCF7WybAEwA6EzXfA9ZVHV14mvH7E23nrVcdtO77Lr3Jt6b3SXtsFJRvUFMPm8gh16ynI7dg7pasWVNBSfds4j3X+3EExcM4LQn5rP3KetY9U5H7v/sELoPqWHgAVuoqCiBv+Ft5OBPLGL16k7MeasP++2/7CPbzz1vKq+/1p8Zr/dPIboiaKU+OEk3A58GlkXEvknZ94H/B7yf7HZJRDyabLsYOIvcpLvfiIjHGjt+Yzf6Htni6HMBdQfuBS6IiLXbOc9EYCJAT/Upqd+IDWsreeWv3TnoyHXtMsHVVcPk8way5wnr2e3YXNOq28417PbJDUgwYPQWEGxeVUGXPnUcesmKD7774KmD6bVbdVqht7mRo5YzbtwiDjroYao61tG1azXf+s7z/Pxn4/jnf3mdXr238J8/OCztMFtH6zY/bwGu46O3nV0TEVc2LEi6uE4DRpGb+ONxSXtFxA5nGC9qGyKZAfhe4PaIuK+Y52otvfrUUFMjNqytpGPnOg48fD13//eAtMNqcxHwf5f0p/ce1ez/lTUflA//xw0seqELg8dtZvW7VdRVi8471VGzSURAVddgwV+6oMpgpz3bT4K75eb9ueXm/QHYb/9l/NMpb/Dzn43j2PFv8/GPL+Hi7xxRNtN856WVElxEPJ208PJxInBnRGwB3pU0BziYRgY9i5bglHuJw03ArIi4uljnaW19BlbzzWvnUVEBFRXw9MO9eOHxnmmH1eaWTu3EWw/2oM/eW7j3M7mu04P+fSV7/9M6/u+S/txz/FAqqoIjfrYMCTatqOTRs3ZGgm4Dazjy5+83cYb24bzzp7JsaVeuvvYJAP767FB+f/uolKNqORV/wsuvSzqD3PuZL4yIVeT68J9vsM8CmujXL2YN7jDgi8BrkqYnZR+0pUvVu7O6cO4n9047jNTtPHYLE958Z7vbjrryo8mrx9AaTn1sQbHDKguvvTqA117N1fo/fdznU46mSPKvwfWT1PAl8hOTbqnGXA/8MDnLD8mNB3yl0BAhv0e1RG7K8t0j4nJJuwA7R8SLjX0vIp6lFCZlN7NWVeAI6fKIGFvI8SNi6Qfnkm4AHkk+LuRvHzIYmpTtUD63Yf4KOAQ4Pfm8DvjvfIM1swwq4pTlkhreS3My8Hqy/hBwmqROknYDRgCNVrTyaaJ+IiIOlPQyQESsktSxGXGbWVa00iCDpDvITebRT9IC4DLgCEljkrPMJZmLMiJmSLobmEnuQYJzGxtBhfwSXLWkyuRkSOpP2bxTx8yKobVu4o2I07dTfFMj+/8I+FG+x88nwf0SuB8YIOlH5GYX+W6+JzCzjIk2GUVtFfm8F/V2SVPJTZkk4KSI8Jvtzdqzkrolf8fyGUXdBdgIPNywLCLmFTMwMythWUlwwB/48OUznYHdgNnkHpcws3aoFB6kz0c+TdT9Gn5OZhL5WtEiMjNrJQU/yRAR0yR9ohjBmFmZyEoNTtK/N/hYARwILCpaRGZW2rI0igr0aLBeQ65P7t7ihGNmZSELNbjkBt8eEfHNNorHzEqcyMAgg6QOEVEjKSOz9JlZqyn3BEfuIdYDgemSHgLuATbUbyyXCSzNrJWVyPsW8pFPH1xnYAW5dzDU3w8XgBOcWXuVgUGGAckI6ut8mNjqlUn+NrNiyEINrhLozvYnrSyTH8/MiqJMMkBjCW5xRFzeZpGYWXkokZc656OxBOfpxs1su7LQRD26zaIws/JS7gkuIla2ZSBmVj6y9KiWmdmHMtIHZ2b2EaJ8Ouid4MyscK7BmVlWZWEU1cxs+5zgzCyTMjbhpZnZ33INzsyyyn1wZpZdTnDNIKFOndKOomTdsN/H0g6h5E169860QyhpBx/bOg8otVYNTtLNwKeBZRGxb1LWB7gLGA7MBT4fEaskCbgW+BS5l9F/KSKmNXb8itYJ08zajSA34WU+S9NuAcZvU3YR8EREjACeSD4DHAeMSJYJwPVNHdwJzswKUv/SmXyWpkTE08C21coTgVuT9VuBkxqU3xY5zwO9JQ1q7PhOcGZWuMhzgX6SpjRYJuRx9IERsThZXwIMTNaHAPMb7LcgKduh0uqDM7OyoMi7E255RIxt7nkiIqTm9/i5Bmdmhcm39tb8gYil9U3P5M9lSflCYFiD/YYmZTvkBGdmBWutPrgdeAg4M1k/E3iwQfkZyhkHrGnQlN0uN1HNrGCt9aiWpDuAI8j11S0ALgN+Ctwt6SzgPeDzye6PkrtFZA6520S+3NTxneDMrHCtdB9cRJy+g00feWVCRARwbiHHd4Izs8Jk7M32ZmZ/ywnOzLKo/kbfcuAEZ2YFU115ZDgnODMrjN+qZWZZ5hl9zSy7XIMzs6zyIIOZZVMA+T9snyonODMrmPvgzCyTfB+cmWVXhJuoZpZdrsGZWXY5wZlZVrkGZ2bZFEBteWQ4JzgzK5hrcGaWXR5FNbOscg3OzLLJ0yWZWVYJkAcZzCyrCnizfaqc4MysMG6ilreTv7KE8ae+TwTMnd2Fq761O9VbK9IOq6T4GsHWzeLCz+5J9dYKamvg749fwxnfWsKSeR358Tm7snZVB0bst5Fv/9c8qjoGyxZU8fMLdmHDmkrq6sRXLlnEwUevS/vHaIbyeRa1aH8jJXWW9KKkVyTNkPSDYp2rNfUduJUTv7SE8z4ziq+O34+KSjjihBVph1VSfI1yqjoFV9zzNr9+fDbXT57NlKd6MGtqV2780SA++//e55a/zqJ771om3dEHgN9fO5DDT1jNrya/ycXXz+W6i4el/BM0nyK/JW3F/Cd3C3BURIwGxgDjJY0r4vlaTWUldOxcR0Vl0KlzLSuWdUw7pJLjawQSdOmWmxitplrUVgsJXnm2B3//6dUAHPO5lTw3qdcH+29cVwnAhrWV9BlYnUrcraJ+RpGmlpQVrYkaEQGsTz5WJUv6P3ETViztyP/esDO//ct0tmyuYNozvZj2TK+0wyopvkYfqq2Frx+7N4vmduSELy1n0K5b6NarlsrkN6vfoGqWL6kC4F8uXMIlp+/BQ7/px+aNFfz0rrdTjLwFonxGUYvaaSKpUtJ0YBkwOSJeKOb5WkP3njUccswqvnT4aL4wbgydu9Zy1EnL0w6rpPgafaiyEq5/fDa3T53J7OldmT+n8w73feqBnTjm8yu5fepMfvjbd7jivF2pK5OZcT8i8lyaIGmupNckTZc0JSnrI2mypLeSP3dqbphFTXARURsRY4ChwMGS9t12H0kTJE2RNKU6NhcznLwc8HdrWTq/E2tWVlFbU8FfHuvDxw5c3/QX2xFfo4/q3quW0YeuZ9bUrmxYU0ltTa58+eIq+u2ca4pOuqMPh5+wGoCRYzeydYtYu7I8x/kUkdeSpyMjYkxEjE0+XwQ8EREjgCeSz83SJsNeEbEaeBIYv51tEyNibESMrdKO//VrK8sWdWSfAzbQqXMtEIw5dA3z304/rlLia5SzekUl69fk+tS2bBLTnu7BsBFbGH3Yep55pDcAk+/pwyHHrgFgwJBqpj/bA4B5b3Vi65YKevWtSSX2FituH9yJwK3J+q3ASc09UNH++ZDUH6iOiNWSugDHAD8r1vlay+zp3Xnmjztx3SMzqK0Rb8/syh/vGJB2WCXF1yhn5dIqrjx/F+rqRF0dHH7CasYds5Zd99rMj8/ZlVuuGMSe+27i2NNXAjDhsoX84pvDuO+G/gj45jXzkNL9GZolgPyb1v3qm56JiRExcZuj/UlSAP+TbBsYEYuT7UuAgc0NVVGkkQ5J+5PLvpXkaop3R8TljX2nZ0XfGNfpuKLEY+3DpHdLvps3VQcfO58pr2xuUVrt1W1wjBt5dl77/mnK96c2aHp+hKQhEbFQ0gBgMnAe8FBE9G6wz6qIaFY/XDFHUV8FDijW8c0sRa00OhIRC5M/l0m6HzgYWCppUEQsljSI3CBls7SvW8/NrOXqm6j5LI2Q1E1Sj/p14JPA68BDwJnJbmcCDzY31PIcwjGzVLXSw/YDgfuV64jsAPw+IiZJegm4W9JZwHvA55t7Aic4MytcKyS4iHgHGL2d8hXA0S0+AU5wZlaw0ngMKx9OcGZWGL9Vy8yyzBNemll2OcGZWSYFUOcEZ2aZ5EEGM8syJzgzy6QAastjIjsnODMrUEA4wZlZVrmJamaZ5FFUM8s01+DMLLOc4MwskyJy70ssA05wZlY41+DMLLOc4Mwsm8KjqGaWUQHhG33NLLP8qJaZZVJEq702sNic4MyscB5kMLOsCtfgzCybPOGlmWWVH7Y3s6wKIPyolpllUnjCSzPLsHAT1cwyq0xqcIoSGg2R9D7wXtpxNNAPWJ52ECXM16dppXaNdo2I/i05gKRJ5H6ufCyPiPEtOV9LlFSCKzWSpkTE2LTjKFW+Pk3zNUpXRdoBmJkVixOcmWWWE1zjJqYdQInz9Wmar1GK3AdnZpnlGpyZZZYTnJlllhPcdki6WdIySa+nHUspkjRM0pOSZkqaIen8tGMqJZI6S3pR0ivJ9flB2jG1V+6D2w5JhwPrgdsiYt+04yk1kgYBgyJimqQewFTgpIiYmXJoJUGSgG4RsV5SFfAscH5EPJ9yaO2Oa3DbERFPAyvTjqNURcTiiJiWrK8DZgFD0o2qdETO+uRjVbK4JpECJzhrEUnDgQOAF1IOpaRIqpQ0HVgGTI4IX58UOMFZs0nqDtwLXBARa9OOp5RERG1EjAGGAgdLcldHCpzgrFmSvqV7gdsj4r604ylVEbEaeBJI7YHz9swJzgqWdKLfBMyKiKvTjqfUSOovqXey3gU4Bngj1aDaKSe47ZB0B/AcsLekBZLOSjumEnMY8EXgKEnTk+VTaQdVQgYBT0p6FXiJXB/cIynH1C75NhEzyyzX4Mwss5zgzCyznODMLLOc4Mwss5zgzCyznODKiKTa5JaM1yXdI6lrC451i6RTkvUbJY1sZN8jJB3ajHPMlfSRty/tqHybfdY3tn07+39f0jcLjdGyzQmuvGyKiDHJDCdbga823CipWe+5jYh/bWImkCOAghOcWdqc4MrXM8CeSe3qGUkPATOTh7x/LuklSa9KOhtyTx9Iuk7SbEmPAwPqDyTpKUljk/XxkqYlc5k9kTxM/1Xg35La498nd+rfm5zjJUmHJd/tK+lPyRxoNwJq6oeQ9ICkqcl3Jmyz7Zqk/AlJ/ZOyPSRNSr7zjKR9WuVqWib5zfZlKKmpHQdMSooOBPaNiHeTJLEmIg6S1An4i6Q/kZvxY29gJDAQmAncvM1x+wM3AIcnx+oTESsl/RpYHxFXJvv9HrgmIp6VtAvwGPAx4DLg2Yi4XNLxQD5PgHwlOUcX4CVJ90bECqAbMCUi/k3S95Jjf53cS1y+GhFvSfoE8CvgqGZcRmsHnODKS5dkCh7I1eBuItd0fDEi3k3KPwnsX9+/BvQCRgCHA3dERC2wSNKft3P8ccDT9ceKiB3NifePwMjcI6kA9ExmFjkc+Gzy3T9IWpXHz/QNSScn68OSWFcAdcBdSfnvgPuScxwK3NPg3J3yOIe1U05w5WVTMgXPB5Jf9A0Ni4DzIuKxbfZrzWdFK4BxEbF5O7HkTdIR5JLlIRGxUdJTQOcd7B7JeVdvew3MdsR9cNnzGHBOMp0RkvaS1A14Gjg16aMbBBy5ne8+Dxwuabfku32S8nVAjwb7/Qk4r/6DpDHJ6tPAPydlxwE7NRFrL2BVktz2IVeDrFcB1NdC/5lc03ct8K6kzyXnkKTRTZzD2jEnuOy5kVz/2jTlXprzP+Rq6vcDbyXbbiM3W8rfiIj3gQnkmoOv8GET8WHg5PpBBuAbwNhkEGMmH47m/oBcgpxBrqk6r4lYJwEdJM0CfkouwdbbQG6iyNfJ9bFdnpR/ATgriW8GcGIe18TaKc8mYmaZ5RqcmWWWE5yZZZYTnJlllhOcmWWWE5yZZZYTnJlllhOcmWXW/weJP9BS3eU1EwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Results for highest accuracy model\n",
      "Trained DeepConvNet on single session using train/test split and got accuracy of: 0.909375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhV0lEQVR4nO3dd5wV9b3/8ddnC30BV4oISJFiEBUVEWvQJIJ6vWBu1HhzjYnGgppYY0sMlmtJohijP82PRKLYNWjEhhqDYldAQKQoSpcinaXunvO5f8wsrrjlnN09O+fMvp8+5rFzZubMfM5x+ey3zPc75u6IiMRRXtQBiIhkihKciMSWEpyIxJYSnIjElhKciMRWQdQBVNS2ON87dcmqkLLKso9bRh1C1rNmTaMOIattLd3AjrItVpdzDD2mpa9Zm0jp2Kkzt7/s7sPqcr26yKps0qlLAeOe2yPqMLLWtT0GRR1C1svfu0/UIWS1dz8fW+dzrF6b4P2Xu6R0bGGnz9tVtc/MmgGTgaYEuegf7j7KzHoAjwO7A1OBM9x9h5k1BcYBBwNrgNPcfWF111cVVUTS5CQ8mdJSg+3Ase5+ADAAGGZmg4HfA3e6ey9gHXB2ePzZwLpw+53hcdVSghORtDiQxFNaqj1PoCR8WRguDhwL/CPc/iAwIlwfHr4m3P89M6u2uq0EJyJpS6b4X03MLN/MpgOrgFeBz4H17l4WHrIU6ByudwaWAIT7NxBUY6uUVW1wIpL9HKe05upnuXZmNqXC6zHuPmbnudwTwAAzaws8A+xTb4GiBCciaXIgUUP1s4LV7j6wxnO6rzezScBhQFszKwhLaV2AZeFhy4CuwFIzKwDaEHQ2VElVVBFJW320wZlZ+7Dkhpk1B34AzAEmAT8KDzsTeDZcnxC+Jtz/b69hthCV4EQkLQ4k6mcWok7Ag2aWT1DYetLdnzez2cDjZva/wEfA/eHx9wMPmdl8YC3w45ouoAQnImlLuQWuGu4+Eziwku1fAN+66dPdtwGnpHMNJTgRSYvj6bTBRUoJTkTS4g6luZHflOBEJF1GgjoNZ20wSnAikhYHkirBiUhcqQQnIrEU3OirBCciMeRAqefGGAElOBFJi2MkcmQQlBKciKQt6aqiikgMqQ1ORGLMSKgNTkTiKJjRVwlORGLI3djh+VGHkRIlOBFJW1JtcCISR0Eng6qoIhJL6mQQkZhSJ4OIxFpCN/qKSBw5RqnnRurIjShFJGuok0FEYssxVVFFJL7UyZDl1n/ZhKcu70nJ6kLMnENO/4ojfr6SL2e34NnfdKdsu5FXAP9540K6DtjM9H/uzuS/dMKBpi0TDL9pIZ36bY36Y0TistGLOfT7m1i/uoDzju0bdThZobAwwR/unERhYZL8fOetyV14ZNy+HDBgFWefN4OCgiTzP9uNP90+kGQyN5JDVdzRbSJmNhb4D2CVu/fP1HVqK6/AOeE3i+ncfwvbS/K456T+9DpyAxNv7cqxFy+j75ANzJvUhom3deWcx+eyW9ftnPPEHJq3STDv9TY8c20PLvjn7Kg/RiReeaKYCX9vx6/vWhJ1KFmjtDSPa64YwrZtBeTnJ7n9T5OYNqUjl135Adf++rssW1bE/5w5i+8ft4hXJvaIOtw6CToZcmOoVibT8APAsAyev05adyilc/8tADRtlaRDr61sXNEEM9heEvzP27Ypn6KOpQB0O7iE5m0SAOx1YAkbVzSJJvAsMOv9Vmxa12gL/1Uwtm0LvpOCgiT5BUmSSaOsLI9ly4oA+GhqR444ammUQdabBHkpLVHL2G+pu082s+6ZOn99Wre0CV/ObkHXASWc+LtF/P3Mvrx0S1c8aZz3j2+X0qY80Z4+313f8IFKVsvLc+6691X27FzC88/2Yt7cYvLznd591vLZp8UcefRS2nfYEnWYdeaYJrzMFds35/HIyN6ceN1imhUlefWODpz428X0P34dM58v5umre3D2w/N2Hv/5u0VMebI95z01J8KoJRslk8Yvzz+Oli138Nsb3qFb943c9r+DOWfkDAoLE3w0dQ8SidxIDDXJhtJZKiKP0szONbMpZjZl/dpEg147UWo8OrI3A4avof+wdQBMe7od+4br+524lqUzWu08fvmc5jxzdQ/OGPMZLXYra9BYJXds3tyEmdM7cPAhK5g7Z3euvPQYLr3o+3w8sx1fhtXVXBY8FzUvpaU6ZtbVzCaZ2Wwz+8TMLg63X29my8xsericUOE915jZfDObZ2ZDa4o18gTn7mPcfaC7D2xb3HANl+7w9FU9aN9rK0f+YsXO7a07lLLg/eCX8PN3WrN7920ArF/WhEdG9uaU0V/Qrue2BotTckPrNttp2XIHAE2aJDjw4JUsXVxEm7bB70pBYYJTTpvHi8/tHWWY9SR4sn0qSw3KgMvdvR8wGLjQzPqF++509wHh8iJAuO/HwL4E7fv3mlm1SaPRVlEXTWnFR8+0Y4++W7j7hH0BOO7XSzn51gU8f2M3kmVGQdMkJ9+yAIB//3lPtqwrYMJ13QDIK4ALJ3wSWfxRuvreRex/WAltist4eMpsHrqjIy8/tnvUYUWquHgrl1/1IXl5jpnz5htd+eD9PTnr3BkMOnQ5eXnOC8/tzYzpHaIOtc6CxwbWvTDi7suB5eH6JjObA3Su5i3DgcfdfTuwwMzmA4OAd6t6g7l7nQOt9MRmjwFDgHbASmCUu99f3Xu+s39TH/fcHhmJJw6u7TEo6hCyXn6/PlGHkNXe/XwsG7Yur1NDYOd92/oFTx6Z0rG/7f/CImB1hU1j3H3MrseFHZKTgf7AZcDPgI3AFIJS3jozuwd4z90fDt9zP/CSu/+jqutnshf19EydW0SilcaNvqvdfWB1B5hZK2A8cIm7bzSz+4CbCAqLNwF3AGfVJs5GW0UVkdoJ5oOrn95gMyskSG6PuPvTAO6+ssL+vwLPhy+XAV0rvL1LuK1KkXcyiEiuCWb0TWWp9ixmBtwPzHH30RW2d6pw2MnArHB9AvBjM2tqZj2A3sAH1V1DJTgRSUtwm0i9lOCOAM4APjaz6eG2a4HTzWxAeKmFwHkA7v6JmT0JzCbogb3Q3au9t0wJTkTSUl9jUd39Lai0rvtiNe+5Gbg51WsowYlI2jRdkojEUjBdUm4MOVOCE5G0abC9iMRSMJuIqqgiEkPBUC0lOBGJJZXgRCTG6mskQ6YpwYlIWtSLKiKxpiqqiMSSnskgIrHlQJlKcCISV6qiikg8uaqoIhJT9TnhZaYpwYlI2lSCE5FYqscJLzNOCU5E0uIYZUl1MohITKkNTkTiyVVFFZGYUhuciMSaEpyIxJJjJNTJICJxpU4GEYklVyeDiMSZK8GJSDxpsL2IxJhKcLWw7OOWXNvz0KjDyFqjF74TdQhZ78ojO0cdQnZLJOt8CndIJOue4MysKzAO6Ehwe90Yd7/LzIqBJ4DuwELgVHdfZ2YG3AWcAGwBfubu06q7Rm709YpIVkliKS01KAMud/d+wGDgQjPrB1wNvObuvYHXwtcAxwO9w+Vc4L6aLqAEJyJpcYIqaipLtedxX15eAnP3TcAcoDMwHHgwPOxBYES4PhwY54H3gLZm1qm6a2RVFVVEckFanQztzGxKhddj3H3Mt85o1h04EHgf6Ojuy8NdKwiqsBAkvyUV3rY03LacKijBiUja3FM+dLW7D6zuADNrBYwHLnH3jUFTW/l13M0s9avtQlVUEUlbfVRRAcyskCC5PeLuT4ebV5ZXPcOfq8Lty4CuFd7eJdxWJSU4EUlL0Iual9JSnbBX9H5gjruPrrBrAnBmuH4m8GyF7T+1wGBgQ4WqbKVURRWRtKVRRa3OEcAZwMdmNj3cdi1wG/CkmZ0NLAJODfe9SHCLyHyC20R+XtMFlOBEJG31caOvu78FVd5L8r1KjnfgwnSuoQQnImlxUmtfywZKcCKStvqpoWaeEpyIpMfB62GoVkNQghORtKmKKiKxVU+9qBlXZYIzs7uppqrt7r/KSEQiktXKx6LmgupKcFOq2ScijZUDuZ7g3P3Biq/NrIW7b8l8SCKS7XKlilrjUC0zO8zMZgNzw9cHmNm9GY9MRLKU4cnUlqilMhb1T8BQYA2Au88Ajs5gTCKS7TzFJWIp9aK6+5KKU5gAicyEIyJZz+PRyVBuiZkdDng4tcnFBDNvikhjlQWls1SkUkU9n2CAa2fgS2AAaQ54FZG4sRSXaNVYgnP31cBPGiAWEckVdX84V4NIpRe1p5k9Z2ZfmdkqM3vWzHo2RHAikoXK74NLZYlYKlXUR4EngU7AnsBTwGOZDEpEspt7akvUUklwLdz9IXcvC5eHgWaZDkxEsliu3yYSPl0a4CUzuxp4nCDk0wimDhaRxioLqp+pqK6TYSpBQiv/JOdV2OfANZkKSkSyW+0f5NewqhuL2qMhAxGRHOEGWTAMKxUpjWQws/5APyq0vbn7uEwFJSJZLtdLcOXMbBQwhCDBvQgcD7wFKMGJNFY5kuBS6UX9EcEjvFa4+8+BA4A2GY1KRLJbrveiVrDV3ZNmVmZmrYFVQNcMxxWZwqZJ7hg/n8KmSfLz4c0X2vDQHZ2iDisS675swqOX9aJkdSEYHHb6So4+awXLPmnBU7/pSdn2PPIKnP+6aQHdBpQw/93WjD23L8VdtgOw37C1DL14acSfomG067CVy6+fQdviHTgw8ZmuTHgiaMY+6dSFnPijRSSTxodvd+Dvd+8TbbB1FYcJLyuYYmZtgb8S9KyWAO/W9CYz60pQje1I8JWMcfe7ah9qwyjdblx56t5s25JPfoEz+pnP+HBSa+ZOaxl1aA0uv8AZ/ttFdOm/mW0ledx50v70OWoDz93WjaEXL+U7x6xn9qS2PH/rXlz4xGwAeh6yiV+MnRtx5A0vkTD+dtd3+HxeG5q3KOOucW/x0Qft2K14B4OPXslFPzmSstJ82uy2PepQ60XO96KWc/cLwtW/mNlEoLW7z0zh3GXA5e4+zcyKgKlm9qq7z65DvA3A2LYlH4CCAie/0LPijuwotO5QSusOpQA0a5Wkw95b2bCiCQZsKwm+o20b82ndsTTCKLPDujXNWLcm6IPbuqWAJQtasXv7bQwbsYSnHtybstLg+9qwrmmUYdafHPk3UWUbnJkdtOsCFAMF4Xq13H25u08L1zcRTLHUub4Cz6S8POfeV+byxMxZfDS5iHkfNb7S267WLmnKstkt6TaghBGjFvLcrd248bCDmHBLd068ctHO4xZOa8Ufh+3PmDP3YcWnzSOMODodOm2hZ9+NzPukLZ332sy+A9Yyeuzb3PaX9+j9nfVRh1cvzFNbajyP2dhwjPusCtuuN7NlZjY9XE6osO8aM5tvZvPMbGhN56+uBHdHNfscOLbm8HcG1R04EHi/kn3nAucCNKNFqqfMqGTSuOC4fWjZuoxR9y+kW9+tLJrXOP+xAmzfnMcDI/sw4ncLaVaU4MU7OjL8uoUccPxapj+/O09ctTcjH5lDl/6bue7taTRtmWT2pLaMPbcv174+PerwG1Sz5mX85rZp/HV0P7ZuLiQv3ylqU8plZx1On34buPrWjzh7xBCyYSqhOqm/NrgHgHv49l0Zd7r77RU3mFk/4MfAvgTj4v9lZn3cvcoJeKsswbn7MdUs6SS3VsB44BJ331jJdca4+0B3H1hIdhXfN28sYMbbrThkyKaoQ4lMotR44Py+HDRiNfsPWwvAlPHtd64fcOIaFs9oBUCzogRNWwbz6PQ7Zj2JUqNkbeN59G5+fpJrfz+NSS/vyTuv7wHAmlXNeGfSHoDx6ey2eNJo3XZHtIHWVao9qCmU4Nx9MrA2xSsPBx539+3uvgCYDwyq7g2p3CZSa+EMwOOBR9z96Uxeq760KS6jZesyAJo0S3LQ0ZtY8nl2Jd6G4g5PXLU3HXptZcgvlu/c3rrDDj5/rzUAn73TmvbdtwGwcVXhzvbKRdNb4W603K2sweOOhnPxdR+zZEEr/vno17OJvftGR/Y/eA0Ae+5VQkFhko3rm0QVZP3J/G0iF5nZzLAKu1u4rTOwpMIxS6mh2Stjf14teIjD/cAcdx+dqevUt+KOpVzxp8Xk5Tl5eTD5uba8/6/GedvfgilFTHm6PZ322cztx+8PwAlXLubU277gnzd0J1FmFDZNcsqtXwAw46XdeefhjuTlO4XNkpxx96dYjtfEUtXvgHV874RlLPisiLsffhOAB+/ty6sTunLJdTP5f49Npqw0j9E37E/OV08BS33Cy3ZmVvEZy2PcfUwN77kPuIkgRd5E0Fx2VroxQgYTHHAEcAbwsZlND7dd6+5ZPRPJgjnNuXBo36jDyAo9D9nE6IWV3xF02fMff2vbUWeu4KgzV2Q6rKw0e0YxJw46odJ9t48a0LDBNITUS2er3X1gWqd2X1m+bmZ/BZ4PXy7jm/fgdgm3VSmVoVpGMGV5T3e/0cz2AvZw9w9qCPIt4vCnSkS+IdUe0lqf36yTu5e3iZwMlPewTgAeNbPRBJ0MvYFq81AqJbh7CWZgPxa4EdhE0K52SPqhi0gs1FMvqpk9RjDWvZ2ZLQVGAUPMbABBOXEh4VRt7v6JmT0JzCa4z/bC6npQIbUEd6i7H2RmH4UXWWdmMWglFZFaq6cSnLufXsnm+6s5/mbg5lTPn0qCKzWzfMKPZGbtyZln6ohIJsRmqBbwZ+AZoIOZ3Uwwu8hvMxqViGQvT6sXNVKpjEV9xMymEkyZZMAId9eT7UUas7iU4MJe0y3AcxW3ufviTAYmIlksLgkOeIGvHz7TDOgBzCMYDyYijVBs2uDcfb+Kr8OZRC6o4nARkayR9kiGcH63QzMRjIjkiLiU4Mzssgov84CDgC8zFpGIZLc49aICRRXWywja5MZnJhwRyQlxKMGFN/gWufsVDRSPiGQ5IwadDGZW4O5lZnZEQwYkIjkg1xMcwSj9g4DpZjYBeArYXL4zVyawFJF6luHZROpTKm1wzYA1BLOJlN8P54ASnEhjFYNOhg5hD+osvk5s5XIkf4tIJsShBJcPtKLySStz5OOJSEbkSAaoLsEtd/cbGywSEckNdX+gTIOpLsFpunERqVQcqqjfa7AoRCS35HqCc/dUH8YqIo1MnIZqiYh8LSZtcCIi32LkTgO9EpyIpE8lOBGJqzj0ooqIVE4JTkRiKWYTXoqIfJNKcCISV7nSBpcXdQAikoM8xaUGZjbWzFaZ2awK24rN7FUz+yz8uVu43czsz2Y238xmhk/4q1Z2leDMsILCqKPIWlfsd1zUIWS9l+a9EHUIWW3Q0A31cp56LME9ANwDjKuw7WrgNXe/zcyuDl9fBRwP9A6XQ4H7wp9VUglORNLjBBNeprLUdCr3ycCuw0KHAw+G6w8CIypsH+eB94C2ZtapuvNnVwlORLJemg+daWdmUyq8HuPuY2p4T0d3Xx6urwA6huudgSUVjlsabltOFZTgRCR9qSe41e4+sNaXcXez2leIVUUVkbSZe0pLLa0sr3qGP1eF25cBXSsc1yXcViUlOBFJT6o9qLXviJgAnBmunwk8W2H7T8Pe1MHAhgpV2UqpiioiaauvXlQzewwYQtBWtxQYBdwGPGlmZwOLgFPDw18ETgDmA1uAn9d0fiU4EUlbfQ3VcvfTq9j1rRnF3d2BC9M5vxKciKQvR0YyKMGJSHpi9mR7EZFvUoITkThK80bfSCnBiUjaLJkbGU4JTkTSo6dqiUicaUZfEYkvleBEJK7UySAi8eRA7QfSNyglOBFJm9rgRCSWdB+ciMSXu6qoIhJfKsGJSHwpwYlIXKkEJyLx5EAiNzKcEpyIpE0lOBGJL/WiikhcqQQnIvGk6ZJEJK4MMHUyiEhc1eGp9Q1KCU5E0qMqam5r2bqMS36/kO59tuLAnb/uwZxpraIOK1KX3vIpg4asY/2aQkaedBAAZ1+5gEOPWUtZqbF8cTNGX9OHzZsaz6/Ujm3G5T/sRemOPBJlcNSJG/jpr1ewYnETbhnZjY3rCui93xauvHsxhU2CjPDGhLY8fMceYE7Pftu45t5FEX+K2tBYVMysGTAZaBpe5x/uPipT16tP549azNQ32nDzyF4UFCZp2jxH5obJoFef7siEh/fkit9/unPbR2+35e93dCeZMM66YgGnnbeEsbf3iDDKhlXY1PnDU5/TvGWSslK4bERvDjl2I+PHtOeH53zFkBHrueuqLkx8rJiTzlzDsi+a8MTdHRj97GcUtU2wfnXu/jHIlV7UvAyeeztwrLsfAAwAhpnZ4Axer160KCpjv0M3MfHxdgCUleaxeWPu/iLWl1lT2rBpwze/h2lv70YyYQDMnV5Euz12RBFaZMygecvgj19ZqZEoNcxgxltFHPUf6wH4wSlreXdiGwBeemR3TvrZaoraJgBo264skrjrRfmMIjUtEcvYv1x3d6AkfFkYLtF/4hrs0XUHG9YUcvntC+jRbyvzP27Bfdfvxfat+VGHltWO+6+VvPFS+6jDaHCJBFw0tC9fLmzCST9bTadu22nZJkF++C+rXadSVq8oBGDpF80AuPQ/e5FMGv9z+QoOOWZTVKHXntdfL6qZLQQ2AQmgzN0Hmlkx8ATQHVgInOru62pz/kyW4DCzfDObDqwCXnX39zN5vfqQn+/06r+Z5x/uwEUn7Mu2LXmcdsHyqMPKaj8+fwmJhDFpQuNLcPn5cN+/5vHI1NnMm96CJfObVXlsIgHLFjTlj+Pnc829i/jTFV0p2ZCjfzg9xSU1x7j7AHcfGL6+GnjN3XsDr4WvayWjCc7dE+4+AOgCDDKz/rseY2bnmtkUM5tS6tsyGU5KVq9owurlTZg3PehUePPFYnr13xJxVNnr+yevZNCQtfzhir4Ed0g1Tq3aJDjg8BLmTG3B5g35JMLa5+rlhbTboxQISnODj9tIQSHssdcOuuy9nWULmkQYde2Ze0pLLQ0HHgzXHwRG1PZEGU1w5dx9PTAJGFbJvjHuPtDdBxZa1X/9Gsq6rwr5ankTuvTcCsCBR2xk8WfNI44qOx181DpO+cVSbhjZj+3bcrQkUgfr1+TvLIFt32pMm1xE197bOeCIEt58vi0Arz5VzGFDNwBw+LANzHw3+MO5YU0+Sz9vSqe9crTdMvU2uHblBZhwOXfXMwGvmNnUCvs6unt5tWkF0LG2YWayF7U9UOru682sOfAD4PeZul59undUN6686wsKC53li5sy+orG0zNYlavumMv+gzbQercyHnrjAx66ey9OO3cphU2S3Pz3WQDMnVHEPaN6RRxpw1m7spDbL96LZNJIJuHok9Yz+Acb6dZnG7eM7MYDf+hEr/5bGXr6WgAGDtnEtDeKOOe7+5CX75xz3Ze0Lk5E/ClqwYHUbyxYXaHqWZkj3X2ZmXUAXjWzud+4lLub1b7P1jxDPR1mtj9B8TKfoKT4pLvfWN17Wuft7oMLv1XIk5A1axp1CFnvpXlvRh1CVhs0dAlTZmyrU1tCm5Z7+uB+56V07CtTrp9aQ4LbycyuJ+iYPAcY4u7LzawT8Lq7961NrJnsRZ0JHJip84tIhJJ1vzfUzFoCee6+KVw/DrgRmACcCdwW/ny2ttfQDV4ikp70qqjV6Qg8Y2YQ5KJH3X2imX0IPGlmZwOLgFNrewElOBFJW30Mtnf3L4ADKtm+BvhenS+AEpyI1EYWjFJIhRKciKQpO4ZhpUIJTkTSo6dqiUicacJLEYkvJTgRiSUHkkpwIhJL6mQQkThTghORWHIgkRvT+CvBiUiaHFwJTkTiSlVUEYkl9aKKSKypBCcisaUEJyKx5B48IiwHKMGJSPpUghOR2FKCE5F4cvWiikhMObhu9BWR2NJQLRGJJfd6eWxgQ1CCE5H0qZNBROLKVYITkXjShJciElcabC8iceWAa6iWiMSSa8JLEYkxVxVVRGIrR0pw5lnUG2JmXwGLoo6jgnbA6qiDyGL6fmqWbd9RN3dvX5cTmNlEgs+VitXuPqwu16uLrEpw2cbMprj7wKjjyFb6fmqm7yhaeVEHICKSKUpwIhJbSnDVGxN1AFlO30/N9B1FSG1wIhJbKsGJSGwpwYlIbCnBVcLMxprZKjObFXUs2cjMuprZJDObbWafmNnFUceUTcysmZl9YGYzwu/nhqhjaqzUBlcJMzsaKAHGuXv/qOPJNmbWCejk7tPMrAiYCoxw99kRh5YVzMyAlu5eYmaFwFvAxe7+XsShNToqwVXC3ScDa6OOI1u5+3J3nxaubwLmAJ2jjSp7eKAkfFkYLipJREAJTurEzLoDBwLvRxxKVjGzfDObDqwCXnV3fT8RUIKTWjOzVsB44BJ33xh1PNnE3RPuPgDoAgwyMzV1REAJTmolbFsaDzzi7k9HHU+2cvf1wCQgsgHnjZkSnKQtbES/H5jj7qOjjifbmFl7M2sbrjcHfgDMjTSoRkoJrhJm9hjwLtDXzJaa2dlRx5RljgDOAI41s+nhckLUQWWRTsAkM5sJfEjQBvd8xDE1SrpNRERiSyU4EYktJTgRiS0lOBGJLSU4EYktJTgRiS0luBxiZonwloxZZvaUmbWow7keMLMfhet/M7N+1Rw7xMwOr8U1FprZt56+VNX2XY4pqW5/Jcdfb2ZXpBujxJsSXG7Z6u4DwhlOdgDnV9xpZrV6zq27/6KGmUCGAGknOJGoKcHlrjeBXmHp6k0zmwDMDgd5/9HMPjSzmWZ2HgSjD8zsHjObZ2b/AjqUn8jMXjezgeH6MDObFs5l9lo4mP584NKw9HhUeKf++PAaH5rZEeF7dzezV8I50P4GWE0fwsz+aWZTw/ecu8u+O8Ptr5lZ+3Db3mY2MXzPm2a2T718mxJLerJ9DgpLascDE8NNBwH93X1BmCQ2uPshZtYUeNvMXiGY8aMv0A/oCMwGxu5y3vbAX4Gjw3MVu/taM/sLUOLut4fHPQrc6e5vmdlewMvAd4BRwFvufqOZnQikMgLkrPAazYEPzWy8u68BWgJT3P1SM/tdeO6LCB7icr67f2ZmhwL3AsfW4muURkAJLrc0D6fggaAEdz9B1fEDd18Qbj8O2L+8fQ1oA/QGjgYec/cE8KWZ/buS8w8GJpefy92rmhPv+0C/YEgqAK3DmUWOBn4YvvcFM1uXwmf6lZmdHK53DWNdAySBJ8LtDwNPh9c4HHiqwrWbpnANaaSU4HLL1nAKnp3Cf+ibK24CfunuL+9yXH2OFc0DBrv7tkpiSZmZDSFIloe5+xYzex1oVsXhHl53/a7fgUhV1AYXPy8DI8PpjDCzPmbWEpgMnBa20XUCjqnkve8BR5tZj/C9xeH2TUBRheNeAX5Z/sLMBoSrk4H/DrcdD+xWQ6xtgHVhctuHoARZLg8oL4X+N0HVdyOwwMxOCa9hZnZADdeQRkwJLn7+RtC+Ns2Ch+b8f4KS+jPAZ+G+cQSzpXyDu38FnEtQHZzB11XE54CTyzsZgF8BA8NOjNl83Zt7A0GC/ISgqrq4hlgnAgVmNge4jSDBlttMMFHkLII2thvD7T8Bzg7j+wQYnsJ3Io2UZhMRkdhSCU5EYksJTkRiSwlORGJLCU5EYksJTkRiSwlORGJLCU5EYuv/AN04wiH/pY5ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "####################################################\n",
    "# MULTI TRIAL DEEPCONVNET MODEL\n",
    "####################################################\n",
    "\n",
    "# Configure\n",
    "retrain_model = False\n",
    "tensorboard_name = \"DeepCN_multisession_C0\"\n",
    "best_model_filename = f\"./saved_variables/6/DeepConvNet/{tensorboard_name}\"\n",
    "\n",
    "\n",
    "# Train on train/test split of data from one session\n",
    "## Note: the model is forced to use GPU, if GPU is not available replace with what is available e.g. /cpu:0\n",
    "if (retrain_model): # Retrain or not\n",
    "    with tf.device('/gpu:0'):\n",
    "        keras_deepcn_model.fit(\n",
    "            x= mne_fixed_window_epochs_train_data,\n",
    "            y= ohe_labels_train,\n",
    "            batch_size= 128, # Default: 32\n",
    "            epochs= 500, # Default: 500 (EEGNet paper)\n",
    "            verbose= 1, # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "            #callbacks= [tensorboard_callback(\"EEGNet_raw_signal_0.5s_100samps_50kernlen_02nr\")], # To be used for TF Board\n",
    "            callbacks= [tensorboard_callback(tensorboard_name),\n",
    "                        lowest_loss_model_save_callback(best_model_filename),\n",
    "                        highest_accuracy_model_save_callback(best_model_filename)],\n",
    "            validation_split= 0.3,\n",
    "            shuffle= True,\n",
    "            sample_weight= None, # Can be interesting due to time series\n",
    "            use_multiprocessing=True, # Done for faster speed\n",
    "            workers= 4 # Done for faster speed\n",
    "            )\n",
    "\n",
    "# Convert labels back to original\n",
    "y_test = ohe.inverse_transform(ohe_labels_test)\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"Results for lowest loss model\")\n",
    "keras_deepcn_model = load_lowest_loss_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_deepcn_model.predict(mne_fixed_window_epochs_test_data)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained DeepConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Get results for best validation loss model\n",
    "print(\"\\n\\nResults for highest accuracy model\")\n",
    "keras_deepcn_model = load_highest_accuracy_model(best_model_filename, custom_objects= {\"square\": EEGModels.square, \"log\": EEGModels.log})\n",
    "\n",
    "y_pred = keras_deepcn_model.predict(mne_fixed_window_epochs_test_data)\n",
    "y_pred = ohe.inverse_transform(y_pred)\n",
    "\n",
    "accuracy =  accuracy_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Trained DeepConvNet on single session using train/test split and got accuracy of: {accuracy}\")\n",
    "ConfusionMatrixDisplay.from_predictions(y_true= y_test, y_pred= y_pred)\n",
    "plt.show()\n",
    "\n",
    "# Remove unused variables\n",
    "del accuracy\n",
    "del best_model_filename\n",
    "del retrain_model\n",
    "del tensorboard_name\n",
    "del y_pred\n",
    "del y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "9abd5d1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAUP\n",
    "####################################################\n",
    "\n",
    "# delete unused variables\n",
    "del keras_deepcn_model\n",
    "del mne_fixed_window_epochs_test_data\n",
    "del mne_fixed_window_epochs_train_data\n",
    "del ohe\n",
    "del ohe_labels_test\n",
    "del ohe_labels_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcc63ebc",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f92ed28e6a5fe026f22555c18fed88052bb861e5576fb72d2ac78e2247fef331"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}