code/experimental-notebooks/1-CLA-dataset-exploration-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "337221df",
   "metadata": {},
   "source": [
    "# CLA dataset exploration notebook\n",
    "\n",
    "In this notebook, we perform some initial data exploration of raw EEG data.\n",
    "\n",
    "This experimental notebook uses a database provided by [Kaya et al.](https://doi.org/10.1038/sdata.2018.211). The CLA dataset in particular. Instructions on where to get the data are available on [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis). These instructions are under `bci-master-thesis/code/data/CLA/README.md`. A python variant of the files will be made available in [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5341c6d",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- Checking requirements\n",
    "    - Correct anaconda environment\n",
    "    - Correct module access\n",
    "    - Correct file access\n",
    "- Loading in data\n",
    "    - Exploring data structure\n",
    "    - Classification labels\n",
    "    - Making MNE object\n",
    "    - Storing MNE objects for all CLA data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "292165d3",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## Checking requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f55ad17",
   "metadata": {},
   "source": [
    "### Correct anaconda environment\n",
    "\n",
    "The `bci-master-thesis` anaconda environment should be active to ensure proper support. Installation instructions are available on [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "334d5fd4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Active environment: bci-master-thesis\n",
      "Correct environment: True\n",
      "\n",
      "Python version: 3.8.10\n",
      "Correct Python version: True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING FOR RIGHT ANACONDA ENVIRONMENT\n",
    "####################################################\n",
    "\n",
    "import os\n",
    "from platform import python_version\n",
    "\n",
    "print(f\"Active environment: {os.environ['CONDA_DEFAULT_ENV']}\")\n",
    "print(f\"Correct environment: {os.environ['CONDA_DEFAULT_ENV'] == 'bci-master-thesis'}\")\n",
    "print(f\"\\nPython version: {python_version()}\")\n",
    "print(f\"Correct Python version: {python_version() == '3.8.10'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22166668",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Correct module access\n",
    "\n",
    "The following codeblock will load in all required modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ab632204",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scipy version (1.8.0 recommended): 1.8.0\n",
      "MNE version (1.0.2 recommended): 1.0.2\n",
      "Numpy version (1.21.5 recommended): 1.21.5\n",
      "\n",
      "\n",
      "... LOADED ALL MODULES ...\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# LOADING MODULES\n",
    "####################################################\n",
    "\n",
    "# Performs IO operations\n",
    "import os\n",
    "import scipy.io; print(f\"Scipy version (1.8.0 recommended): {scipy.__version__}\")\n",
    "\n",
    "# Modules tailored for EEG data\n",
    "import mne; print(f\"MNE version (1.0.2 recommended): {mne.__version__}\")\n",
    "\n",
    "# Data manipulation modules\n",
    "import numpy as np; print(f\"Numpy version (1.21.5 recommended): {np.__version__}\")\n",
    "\n",
    "# Datetime object\n",
    "import datetime\n",
    "import pytz\n",
    "\n",
    "# Print progress\n",
    "print(\"\\n\\n... LOADED ALL MODULES ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813bb5de",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Correct file access\n",
    "\n",
    "As mentioned, this experimental notebook uses a database provided by [Kaya et al](https://doi.org/10.1038/sdata.2018.211). The CLA dataset in particular. Instructions on where to get the data are available on [the GitHub repository of the BCI master thesis project](https://www.github.com/pikawika/bci-master-thesis). These instructions are under `bci-master-thesis/code/data/CLA/README.md`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "caa1d182",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All files are available\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHEKCING FILE AVAILABILITY\n",
    "####################################################\n",
    "\n",
    "# You can specify the data directory here, per default it is \"../data/CLA/\"\n",
    "data_directory = r'../data/CLA/'\n",
    "\n",
    "# Files needed for this notebook\n",
    "filenames = [\"CLASubjectA1601083StLRHand.mat\",\n",
    "             \"CLASubjectB1510193StLRHand.mat\",\n",
    "             \"CLASubjectB1510203StLRHand.mat\",\n",
    "             \"CLASubjectB1512153StLRHand.mat\",\n",
    "             \"CLASubjectC1511263StLRHand.mat\",\n",
    "             \"CLASubjectC1512163StLRHand.mat\",\n",
    "             \"CLASubjectC1512233StLRHand.mat\",\n",
    "             \"CLASubjectD1511253StLRHand.mat\",\n",
    "             \"CLASubjectE1512253StLRHand.mat\",\n",
    "             \"CLASubjectE1601193StLRHand.mat\",\n",
    "             \"CLASubjectE1601223StLRHand.mat\",\n",
    "             \"CLASubjectF1509163StLRHand.mat\",\n",
    "             \"CLASubjectF1509173StLRHand.mat\",\n",
    "             \"CLASubjectF1509283StLRHand.mat\"]\n",
    "\n",
    "# Append data directory to filenames\n",
    "filenames = [data_directory + filename for filename in filenames]\n",
    "\n",
    "# Check if all files are available, if not display file name of missing file\n",
    "all_files_available = True\n",
    "\n",
    "for filename in filenames:\n",
    "    if (not os.path.isfile(filename)):\n",
    "        print(filename + \" not available!\")\n",
    "        all_files_available = False\n",
    "\n",
    "# Display succes message if all files are available\n",
    "if (all_files_available):\n",
    "    print(\"All files are available\")\n",
    "    \n",
    "# Cleaning up redundant variables from this codeblock\n",
    "del all_files_available\n",
    "del data_directory\n",
    "del filename"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a71dde",
   "metadata": {},
   "source": [
    "<hr><hr>\n",
    "\n",
    "## Loading in data\n",
    "\n",
    "In this step, we load in the data. The data is provided as `.mat` files and thus originally meant for use with MATLAB. However, thanks to scipy we can get them to work in Python as well. Scipy will load in the `.mat` file as a dictionary. From the article by [Kaya et al.](https://doi.org/10.1038/sdata.2018.211) we know:\n",
    "\n",
    "> The data in each file are represented as an instance of a Matlab structure named `o`, having the following key fields `id`, `nS`, `sampFreq`, `marker` and `data`.\n",
    "\n",
    "We will explore 1 file to check out how these files are structured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "399f5c6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File structure of first mat file: dict_keys(['__header__', '__version__', '__globals__', 'o'])\n",
      "Fieldnames of o structure: ['id', 'tag', 'sampFreq', 'nS', 'marker', 'data', 'chnames', 'binsuV']\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# LOADING 1 FILE AND CHECKING CONTENTS\n",
    "####################################################\n",
    "\n",
    "# Load in the first file\n",
    "first_file_full = scipy.io.loadmat(filenames[0], struct_as_record=False, squeeze_me=True)\n",
    "\n",
    "# show keys of the dictionary\n",
    "print(f\"File structure of first mat file: {first_file_full.keys()}\")\n",
    "\n",
    "# The data is stored inside the matlab structure named \"o\"\n",
    "first_file_o = first_file_full['o']\n",
    "\n",
    "# show fieldnames of the o structure\n",
    "print(f\"Fieldnames of o structure: {first_file_o._fieldnames}\")\n",
    "\n",
    "# Cleaning up redundant variables from this codeblock\n",
    "del first_file_full;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145879be",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Exploring data structure\n",
    "\n",
    "From loading in the data we see that the following data is available:\n",
    "\n",
    "- id: A unique alphanumeric identifier of the record\n",
    "- tag: Unknown field\n",
    "   - Was not specified in the article\n",
    "- sampFreq: Sampling frequency of the EEG data\n",
    "- nS: Number of EEG data samples\n",
    "- marker: The eGUI interaction record of the recording session\n",
    "- data: The raw EEG data of the recording session\n",
    "- chnames: Probably channel names of the EEG data sensors/channels in 10/20 configuration\n",
    "   - Was not specified in article\n",
    "- binsuV: Probably bins per microvolt\n",
    "   - Was not specified in the article\n",
    "\n",
    "We will analyse this data for the first file now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d242a35a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id: 201601081851.951FEF1D\n",
      "\n",
      "tag: NK-data import (auto)\n",
      "\n",
      "sampFreq: 200\n",
      "\n",
      "nS: 671600\n",
      "\n",
      "marker: [0 0 0 ... 0 0 0]\n",
      "marker shape: (671600,)\n",
      "\n",
      "data: [[ -0.    -0.    -0.   ...  -0.    -0.    -0.  ]\n",
      " [ -0.    -0.    -0.   ...  -0.    -0.    -0.  ]\n",
      " [ -0.    -0.    -0.   ...  -0.    -0.    -0.  ]\n",
      " ...\n",
      " [ 23.8  -28.4    4.31 ...  -8.31  -6.    -0.23]\n",
      " [ 10.74 -37.39   5.51 ...  -9.34  -5.99  -0.16]\n",
      " [  0.76 -47.95   3.66 ...  -7.32  -4.9   -0.41]]\n",
      "data shape: (671600, 22)\n",
      "\n",
      "chnames: ['Fp1' 'Fp2' 'F3' 'F4' 'C3' 'C4' 'P3' 'P4' 'O1' 'O2' 'A1' 'A2' 'F7' 'F8'\n",
      " 'T3' 'T4' 'T5' 'T6' 'Fz' 'Cz' 'Pz' 'X5']\n",
      "chnames shape: (22,)\n",
      "\n",
      "binsuV: 1\n",
      "\n",
      "\n",
      "Amount of channel names correspond with amount of channels available (22): True\n",
      "Number of samples corresponds with amount of data records (671600): True\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING DATA CONTENTS OF FIRST FILE\n",
    "####################################################\n",
    "\n",
    "print(\"id: \" + str(first_file_o.id))\n",
    "\n",
    "print(\"\\ntag: \" + str(first_file_o.tag))\n",
    "\n",
    "print(\"\\nsampFreq: \" + str(first_file_o.sampFreq))\n",
    "\n",
    "print(\"\\nnS: \" + str(first_file_o.nS))\n",
    "\n",
    "print(\"\\nmarker: \" + str(first_file_o.marker))\n",
    "print(\"marker shape: \" + str(first_file_o.marker.shape))\n",
    "\n",
    "print(\"\\ndata: \" + str(first_file_o.data))\n",
    "print(\"data shape: \" + str(first_file_o.data.shape))\n",
    "\n",
    "print(\"\\nchnames: \" + str(first_file_o.chnames))\n",
    "print(\"chnames shape: \" + str(first_file_o.chnames.shape))\n",
    "\n",
    "print(\"\\nbinsuV: \" + str(first_file_o.binsuV))\n",
    "\n",
    "# Validating right output\n",
    "print(\"\\n\\nAmount of channel names correspond with amount of channels available (\" + str(first_file_o.data.shape[1]) +\"): \" + str(first_file_o.chnames.size == first_file_o.data.shape[1]))\n",
    "print(\"Number of samples corresponds with amount of data records (\" + str(first_file_o.data.shape[0]) +\"): \" + str(first_file_o.nS == first_file_o.data.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3279ff20",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Classification labels\n",
    "\n",
    "From the article by [Kaya et al.](https://doi.org/10.1038/sdata.2018.211) we know:\n",
    "\n",
    ">The `marker` field contains the recording sessions’ interaction record. This record is in the form of 1D\n",
    "Matlab array of size nSx1, which contains integer values from 0 to 99. Each value encodes the state of the\n",
    "eGUI at the time mapping to the corresponding EEG data sample in the “data” array at the same timeindex location.\n",
    "\n",
    "We see the following codes in the CLA datasets:\n",
    "- 0: “blank” or nothing is displayed in eGUI\n",
    "    - Can be seen as a break between stimuli, thus random EEG data that should probably be ignored\n",
    "- 1: Left hand action\n",
    "    - EEG data for MI of the left hand\n",
    "- 2: Right hand action\n",
    "    - EEG data for MI of the right hand\n",
    "- 3: Passive/neutral\n",
    "    - EEG data for forced relax state, no MI activity\n",
    "- 91: inter-session rest break period\n",
    "- 92: experiment end\n",
    "- 99: initial relaxation period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "40ae430c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "labels present in first file: {0: 476168, 1: 61490, 2: 69202, 3: 64740}\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING LABELS FOR FIRST FILE\n",
    "####################################################\n",
    "\n",
    "unique, counts = np.unique(first_file_o.marker, return_counts=True)\n",
    "print(\"labels present in first file: \" + str(dict(zip(unique, counts))))\n",
    "\n",
    "# Cleaning up redundant variables from this codeblock\n",
    "del unique\n",
    "del counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f660ecca",
   "metadata": {},
   "source": [
    "It was known from the paper by [Kaya et al.](https://doi.org/10.1038/sdata.2018.211) that \n",
    "\n",
    "> CLA-SubjectF-150916-3St-LRHand is one of our earliest BCI experiments and for that reason contains only two BCI signals - left and right hand movements, without passive imagery.\n",
    "\n",
    "To validate the presence of all three signals in all data, we loop the unique counts for all files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e49fbaec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "labels present in ../data/CLA/CLASubjectA1601083StLRHand.mat: {0: 476168, 1: 61490, 2: 69202, 3: 64740}\n",
      "labels present in ../data/CLA/CLASubjectB1510193StLRHand.mat: {0: 534120, 1: 65596, 2: 63935, 3: 65749}\n",
      "labels present in ../data/CLA/CLASubjectB1510203StLRHand.mat: {0: 472267, 1: 65511, 2: 63892, 3: 65730}\n",
      "labels present in ../data/CLA/CLASubjectB1512153StLRHand.mat: {0: 472328, 1: 65574, 2: 63953, 3: 65945}\n",
      "labels present in ../data/CLA/CLASubjectC1511263StLRHand.mat: {0: 471330, 1: 65585, 2: 63945, 3: 65940}\n",
      "labels present in ../data/CLA/CLASubjectC1512163StLRHand.mat: {0: 485680, 1: 65646, 2: 63928, 3: 65946}\n",
      "labels present in ../data/CLA/CLASubjectC1512233StLRHand.mat: {0: 473299, 1: 65373, 2: 63931, 3: 65971, 99: 826}\n",
      "labels present in ../data/CLA/CLASubjectD1511253StLRHand.mat: {0: 472424, 1: 65620, 2: 63757, 3: 65799}\n",
      "labels present in ../data/CLA/CLASubjectE1512253StLRHand.mat: {0: 468751, 1: 65352, 2: 63946, 3: 65951}\n",
      "labels present in ../data/CLA/CLASubjectE1601193StLRHand.mat: {0: 420051, 1: 65158, 2: 63949, 3: 65990, 91: 48014, 92: 207, 99: 1031}\n",
      "labels present in ../data/CLA/CLASubjectE1601223StLRHand.mat: {0: 423278, 1: 65557, 2: 63762, 3: 65151, 91: 48012, 92: 207, 99: 1033}\n",
      "labels present in ../data/CLA/CLASubjectF1509163StLRHand.mat: {0: 410424, 1: 97958, 2: 99169, 90: 9005, 91: 72012, 92: 207, 99: 825}\n",
      "labels present in ../data/CLA/CLASubjectF1509173StLRHand.mat: {0: 393015, 1: 63033, 2: 71268, 3: 63236, 90: 1005, 91: 72012, 92: 207, 99: 824}\n",
      "labels present in ../data/CLA/CLASubjectF1509283StLRHand.mat: {0: 392290, 1: 65848, 2: 64605, 3: 65215, 90: 30005, 91: 48011, 92: 207, 99: 619}\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING LABELS FOR ALL FILES\n",
    "####################################################\n",
    "\n",
    "for filename in filenames:\n",
    "    # Load in file o structure\n",
    "    file = scipy.io.loadmat(filename, struct_as_record=False, squeeze_me=True)['o']\n",
    "    \n",
    "    # Print count\n",
    "    unique, counts = np.unique(file.marker, return_counts=True)\n",
    "    print(f\"labels present in {filename}: \" + str(dict(zip(unique, counts))))\n",
    "    \n",
    "# Cleaning up redundant variables from this codeblock\n",
    "del unique\n",
    "del counts\n",
    "del file\n",
    "del filename"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3223c637",
   "metadata": {},
   "source": [
    "We notice that `CLASubjectF1509163StLRHand.mat` does indeed not contain the 3 marker.\n",
    "We notice some files do contain the `91`,  `92` and `99` marker whilst others don't."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bcbe994",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Making MNE object\n",
    "\n",
    "Having access to all of the data, we can manually make MNE-Python data structures per specification of the [MNE documentation](https://mne.tools/dev/auto_tutorials/simulation/10_array_objs.html). This consist of first making an `MNE info` object which can then be used to create a `raw MNE` object."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98703a9b",
   "metadata": {},
   "source": [
    "#### Making the info object\n",
    "\n",
    "From the article by [Kaya et al.](https://doi.org/10.1038/sdata.2018.211) we know that:\n",
    "> The “data” array describes the measured voltage time-series from 19 EEG leads in 10/20 configuration, two ground leads A1-A2, and one synchronization channel X3, as detailed previously in this document\n",
    "\n",
    "<center>\n",
    "<div>\n",
    "    <img src=\"../images/10-20.svg\" width=\"300\"/>\n",
    "    <br>\n",
    "    <small>By トマトン124 (talk) - Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=10489987</small>\n",
    "</div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9bfebea7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Fp1', 'Fp2', 'F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'A1',\n",
       "       'A2', 'F7', 'F8', 'T3', 'T4', 'T5', 'T6', 'Fz', 'Cz', 'Pz', 'X5'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "####################################################\n",
    "# CHECKING CHANNEL NAMES\n",
    "####################################################\n",
    "\n",
    "# Check the present channels, and conclude X5 is not in 10/20 convention but remember it is the synchronization channel\n",
    "first_file_o.chnames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "770949d3",
   "metadata": {},
   "source": [
    "Using the data we have at our disposal from the article by [Kaya et al.](https://doi.org/10.1038/sdata.2018.211), we make meta data list for each CLA file containing:\n",
    "\n",
    "- File location: String\n",
    "- Subject ID: String\n",
    "- Date of recording: Date\n",
    "- Sex of subject: String\n",
    "- Age of string (highest value of age range): Int\n",
    "- Health condition: String\n",
    "- Prior BCI experience: Bool\n",
    "- BCI literacy level: String"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6aaced4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... LOADED MNE META DATA FOR ALL CLA DATASETS ...\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# META DATA FOR ALL CLA DATA FILES\n",
    "####################################################\n",
    "\n",
    "# Class to store data information\n",
    "class MneDataInfo:\n",
    "    def __init__(self, file_location, subject, date_of_recording, sex, age, health_condition, prior_bci_experience, bci_literacy):\n",
    "        self.file_location = file_location\n",
    "        self.subject = subject\n",
    "        self.date_of_recording = date_of_recording\n",
    "        self.sex = sex\n",
    "        self.age = age\n",
    "        self.health_condition = health_condition\n",
    "        self.prior_bci_experience= prior_bci_experience\n",
    "        self.bci_literacy = bci_literacy\n",
    "    \n",
    "# Create data information for each usefull record\n",
    "meta_data_list = [\n",
    "    # CLASubjectA1601083StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectA1601083StLRHand.mat\",\n",
    "                subject= \"A\",\n",
    "                date_of_recording= datetime.datetime(2016, 1, 8, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - High\"),\n",
    "    \n",
    "    # CLASubjectB1510193StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectB1510193StLRHand.mat\",\n",
    "                subject= \"B\",\n",
    "                date_of_recording= datetime.datetime(2015, 10, 19, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectB1510203StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectB1510203StLRHand.mat\",\n",
    "                subject= \"B\",\n",
    "                date_of_recording= datetime.datetime(2015, 10, 20, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectB1512153StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectB1512153StLRHand.mat\",\n",
    "                subject= \"B\",\n",
    "                date_of_recording= datetime.datetime(2015, 12, 15, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectC1511263StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectC1511263StLRHand.mat\",\n",
    "                subject= \"C\",\n",
    "                date_of_recording= datetime.datetime(2015, 11, 26, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 30,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - High\"),\n",
    "    \n",
    "    # CLASubjectC1512163StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectC1512163StLRHand.mat\",\n",
    "                subject= \"C\",\n",
    "                date_of_recording= datetime.datetime(2015, 12, 16, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 30,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - High\"),\n",
    "    \n",
    "    # CLASubjectC1512233StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectC1512233StLRHand.mat\",\n",
    "                subject= \"C\",\n",
    "                date_of_recording= datetime.datetime(2015, 12, 23, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 30,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - High\"),\n",
    "    \n",
    "    # CLASubjectD1511253StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectD1511253StLRHand.mat\",\n",
    "                subject= \"D\",\n",
    "                date_of_recording= datetime.datetime(2015, 11, 25, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 30,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectE1512253StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectE1512253StLRHand.mat\",\n",
    "                subject= \"E\",\n",
    "                date_of_recording= datetime.datetime(2015, 12, 25, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"F\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectE1601193StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectE1601193StLRHand.mat\",\n",
    "                subject= \"E\",\n",
    "                date_of_recording= datetime.datetime(2016, 1, 19, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"F\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectE1601223StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectE1601223StLRHand.mat\",\n",
    "                subject= \"E\",\n",
    "                date_of_recording= datetime.datetime(2016, 1, 22, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"F\",\n",
    "                age= 25,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectF1509163StLRHand.mat - removed due to only 2 recorded signals\n",
    "    \n",
    "    # CLASubjectF1509173StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectF1509173StLRHand.mat\",\n",
    "                subject= \"F\",\n",
    "                date_of_recording= datetime.datetime(2015, 9, 17, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 35,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "    \n",
    "    # CLASubjectF1509283StLRHand.mat\n",
    "    MneDataInfo(file_location= \"../data/CLA/CLASubjectF1509283StLRHand.mat\",\n",
    "                subject= \"F\",\n",
    "                date_of_recording= datetime.datetime(2015, 9, 28, 0, 0, 0, 0, pytz.UTC),\n",
    "                sex= \"M\",\n",
    "                age= 35,\n",
    "                health_condition= \"Healthy\",\n",
    "                prior_bci_experience = False,\n",
    "                bci_literacy= \"Intermediate - Low\"),\n",
    "]\n",
    "\n",
    "print(\"... LOADED MNE META DATA FOR ALL CLA DATASETS ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf37e2b",
   "metadata": {},
   "source": [
    "We now create MNE-Python data structures from scratch, based upon the [MNE documentation](https://mne.tools/stable/auto_tutorials/simulation/10_array_objs.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8d8e0365",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating RawArray with float64 data, n_channels=21, n_times=671600\n",
      "    Range : 0 ... 671599 =      0.000 ...  3357.995 secs\n",
      "Ready.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"table table-hover table-striped table-sm table-responsive small\">\n",
       "    <tr>\n",
       "        <th>Measurement date</th>\n",
       "        \n",
       "        <td>January 08, 2016  00:00:00 GMT</td>\n",
       "        \n",
       "    </tr>\n",
       "    <tr>\n",
       "        <th>Experimenter</th>\n",
       "        \n",
       "        <td>Kaya et al.</td>\n",
       "        \n",
       "    </tr>\n",
       "        <th>Participant</th>\n",
       "        \n",
       "            \n",
       "            <td>A</td>\n",
       "            \n",
       "        \n",
       "    </tr>\n",
       "    <tr>\n",
       "        <th>Digitized points</th>\n",
       "        \n",
       "        <td>0 points</td>\n",
       "        \n",
       "    </tr>\n",
       "    <tr>\n",
       "        <th>Good channels</th>\n",
       "        <td>21 EEG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <th>Bad channels</th>\n",
       "        <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <th>EOG channels</th>\n",
       "        <td>Not available</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <th>ECG channels</th>\n",
       "        <td>Not available</td>\n",
       "    \n",
       "    <tr>\n",
       "        <th>Sampling frequency</th>\n",
       "        <td>200.00 Hz</td>\n",
       "    </tr>\n",
       "    \n",
       "    \n",
       "    <tr>\n",
       "        <th>Highpass</th>\n",
       "        <td>0.00 Hz</td>\n",
       "    </tr>\n",
       "    \n",
       "    \n",
       "    <tr>\n",
       "        <th>Lowpass</th>\n",
       "        <td>100.00 Hz</td>\n",
       "    </tr>\n",
       "    \n",
       "    \n",
       "    \n",
       "    <tr>\n",
       "        <th>Duration</th>\n",
       "        <td>00:55:57 (HH:MM:SS)</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<RawArray | 21 x 671600 (3358.0 s), ~107.6 MB, data loaded>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE MNE INFO OBJECTS\n",
    "####################################################\n",
    "\n",
    "def create_mne_raw(data_info: MneDataInfo):\n",
    "    # Get file as matlab file\n",
    "    file = scipy.io.loadmat(data_info.file_location,\n",
    "                            struct_as_record=False,\n",
    "                            squeeze_me=True)\n",
    "    \n",
    "    # Keep only the matlab \"o\" structure\n",
    "    file = file[\"o\"]\n",
    "    \n",
    "    # Get channel names but drop last sync channel\n",
    "    ch_names = file.chnames[:-1].tolist()\n",
    "    \n",
    "    # All channels are EEG\n",
    "    ch_types = [\"eeg\"] * len(ch_names)\n",
    "    \n",
    "    # Sample frequency is stored\n",
    "    sfreq = file.sampFreq\n",
    "    \n",
    "    # Create an MNE info object\n",
    "    mne_info = mne.create_info(ch_names= ch_names,\n",
    "                               ch_types= ch_types,\n",
    "                               sfreq= sfreq)\n",
    "    \n",
    "    # Update MNE info object further    \n",
    "    mne_info['description'] = f\"Data from {data_info.file_location} | age {data_info.age}| health {data_info.health_condition} | prior BCI experience {data_info.prior_bci_experience} | BCI literacy {data_info.bci_literacy}\"\n",
    "    \n",
    "    mne_info.set_montage('standard_1020')\n",
    "    \n",
    "    mne_info['experimenter'] = 'Kaya et al.'\n",
    "    \n",
    "    if (data_info.sex == \"m\"):\n",
    "        sex = 1\n",
    "    elif (data_info.sex == \"f\"):\n",
    "        sex = 2\n",
    "    else:\n",
    "        sex = 0\n",
    "    \n",
    "    mne_info['subject_info'] = {\n",
    "        \"his_id\": data_info.subject,\n",
    "        \"sex\": sex\n",
    "    }\n",
    "    \n",
    "    # Get data in correct format\n",
    "    data = file.data # RAW data\n",
    "    data = data[:, :-1] # Remove last sync channel\n",
    "    data = data.transpose() # Right order\n",
    "    data = data * 10e-6  # From microvolts to volts\n",
    "    \n",
    "    # Create raw object\n",
    "    mne_raw = mne.io.RawArray(data, mne_info)\n",
    "    \n",
    "    # Set date\n",
    "    mne_raw.set_meas_date(data_info.date_of_recording.astimezone(datetime.timezone.utc))\n",
    "    \n",
    "    # Return created mne info\n",
    "    return mne_raw\n",
    "\n",
    "first_raw_mne = create_mne_raw(meta_data_list[0])\n",
    "first_raw_mne\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4beef4",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Storing MNE objects for all CLA data\n",
    "\n",
    "We now store the CLA matlab dataset to MNE files (FIF) to be re-used later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fa8547b5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating RawArray with float64 data, n_channels=21, n_times=671600\n",
      "    Range : 0 ... 671599 =      0.000 ...  3357.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectA1601083StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectA1601083StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=729400\n",
      "    Range : 0 ... 729399 =      0.000 ...  3646.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectB1510193StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectB1510193StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=667400\n",
      "    Range : 0 ... 667399 =      0.000 ...  3336.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectB1510203StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectB1510203StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=667800\n",
      "    Range : 0 ... 667799 =      0.000 ...  3338.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectB1512153StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectB1512153StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=666800\n",
      "    Range : 0 ... 666799 =      0.000 ...  3333.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectC1511263StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectC1511263StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=681200\n",
      "    Range : 0 ... 681199 =      0.000 ...  3405.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectC1512163StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectC1512163StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=669400\n",
      "    Range : 0 ... 669399 =      0.000 ...  3346.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectC1512233StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectC1512233StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=667600\n",
      "    Range : 0 ... 667599 =      0.000 ...  3337.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectD1511253StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectD1511253StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=664000\n",
      "    Range : 0 ... 663999 =      0.000 ...  3319.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectE1512253StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectE1512253StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=664400\n",
      "    Range : 0 ... 664399 =      0.000 ...  3321.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectE1601193StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectE1601193StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=667000\n",
      "    Range : 0 ... 666999 =      0.000 ...  3334.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectE1601223StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectE1601223StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=664600\n",
      "    Range : 0 ... 664599 =      0.000 ...  3322.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectF1509173StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectF1509173StLRHand_raw.fif\n",
      "[done]\n",
      "Creating RawArray with float64 data, n_channels=21, n_times=666800\n",
      "    Range : 0 ... 666799 =      0.000 ...  3333.995 secs\n",
      "Ready.\n",
      "Overwriting existing file.\n",
      "Writing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectF1509283StLRHand_raw.fif\n",
      "Closing /Users/lennertbontinck/Documents/GitHub/VUB-BCI-thesis/code/experimental-notebooks/../data/CLA/CLASubjectF1509283StLRHand_raw.fif\n",
      "[done]\n"
     ]
    }
   ],
   "source": [
    "####################################################\n",
    "# CREATE FIF FILES FOR ALL CLA DATASET\n",
    "####################################################\n",
    "\n",
    "for meta_data in meta_data_list:\n",
    "    mne_file = create_mne_raw(meta_data)\n",
    "    filename = meta_data.file_location[:-4] + \"_raw.fif\"\n",
    "    mne_file.save(filename, overwrite=True)\n",
    "    \n",
    "# Clear unused variables\n",
    "del meta_data\n",
    "del mne_file\n",
    "del filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b4154ae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "# CLEAR VARIABLES\n",
    "####################################################\n",
    "\n",
    "# End of notebook, clear all variables\n",
    "del filenames\n",
    "del first_file_o\n",
    "del first_raw_mne\n",
    "del meta_data_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87340a14",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}