Skip to content

Latest commit

 

History

History
84 lines (52 loc) · 3.2 KB

README.md

File metadata and controls

84 lines (52 loc) · 3.2 KB
Raw

Decoding Thoughts

CI codecov license black Documentation Status

Investigating the difference in robustness between decoded brain (fMRI) representations versus those learned by a convolutional neural network using methods from computational neuroscience and deep learning.

decoding-thoughts

Installation

Clone the repository.

git clone https://github.com/theairbend3r/decoding-thoughts.git

Create environment.

conda env create -f environment.yml

Install dependencies.

pip install -r requirements.txt

Install package.

pip install -e . # or make install

Content

The main package resides in ./src/.

Jupyter notebooks that use modules from the above package are in ./notebooks/.

  1. Exploratory Analysis
  2. Stimulus Classifier
  3. fMRI Classifier
  4. Representational Similarity Matrix and Robustness

Experiment and Analysis

Goal

  • Study the robustness of CNN models to increasing noise and compare the similarity of its learned embeddings to brain's representation of the same visual stimulus.

Hypothesis

  • Latent representations generated by the most robust CNN (Stimulus) model may be closest to the one generated by the DNN (fMRI/Brain) model.

Dataset

  • The kay dataset (Kay et. al., 2008) contains images and the corresponding brain fMRI readings collected from 2 human subjects who were shown a set of natural images while their brain activity was being recorded.
  • Data shapes:
    • fMRI: 8428 voxels
    • Images: (128 x 128) pixels

Method

  • Train a 3 layer DNN to predict output labels using fMRI data.
  • Finetune 5 CNNs (pretrained on ImageNet) to predict output labels using images.
  • Extract embeddings for all images in test set from the trained models (CNN and DNN).
  • Perform representation similarity analysis using cosine-similarity metric for each model using its learned embeddings.
  • Add increasing levels of gaussian blurring to the images as they are passed through these networks and capture the accuracy.
  • Compute the correlation between the representational similary matrix (RSM) (computed above) for fMRI (DNN) vs Stimulus (CNN1, CNN2, ..., CNN5) models.
  • Observe the relationship between correlation coefficients calculated betwee RSMs (fMRI vs Stimulus) and the CNN model (Stimulus) that drops the most accuracy for noisy input.

Results

  • See notebooks in the ./notebooks/ directory.

Meta

Akshaj Verma – @theairbend3r

Distributed under the GNU GPL-V3 license. See LICENSE for more information.

https://github.com/theairbend3r/decoding-thoughts