Updated images, fixed comments and small rewriting (#154)

* Rename Learning_delays_v01.ipynb to Learning_delays.ipynb

Rename file

* Delete Learning_delays.ipynb

* Add files via upload

I have collected most of the delay related functions and variables to a DelayLayer class. Applied surrogate update to delays as suggested by (Tomas Fiers, Markus Ghosh and Alessandro Galloni). Method B was applied. The delay is applied only to the first layer. Appling the delay with Method B subtracts a duration=delay_max from the input duration. So every subsequent push through a layer decreases the duration(or effective duration). So, for applying multiple delay layers, the duration of the inputs needs to be appended with delay_max*number_delay_layer.

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* Update Learning_delays.ipynb

* 'Major_delay_learning_update'

* Final delay update

* MD file

* bib

* Sync

* Modified_writing_im_size

* changed to TIFF

---------

Co-authored-by: Karim Habashy <[email protected]>

paper/sections/delays/Delays.bib

@@ -0,0 +1,138 @@
+@article{LAJ1948,
+	author = {L. A. Jeffress},
+	title = {A place theory of sound localization},
+	journal = {J. Comp. Physiol.},
+	volume = "41",
+	number = "",
+	pages = "35–39",
+	year = "1948",
+	DOI = "https://doi.org/10.1037/h0061495"
+}
+
+@article{KLWH2001,
+	author = {Richard Kempter and Christian Leibold and Hermann Wagner and and J. Leo van Hemmen},
+	title = {Formation of temporal-feature maps by axonal propagation of synaptic learning},
+	journal = {J. Comp. Physiol.},
+	volume = "98",
+	number = "",
+	pages = "4166-71",
+	year = "2001",
+	DOI = "https://doi.org/10.1073/pnas.061369698"
+}
+
+@article{EMI2006,
+	author = {Eugene M Izhikevich},
+	title = {Polychronization: computation with spikes},
+	journal = {Neural Comput.},
+	volume = "18",
+	number = "",
+	pages = "245-82",
+	year = "2006",
+	DOI = "https://doi.org/10.1162/089976606775093882"
+}
+
+@article{JSZK2015,
+	author = {Max Jaderberg and Karen Simonyan and Andrew Zisserman and Koray Kavukcuoglu},
+	title = {Spatial Transformer Networks},
+	journal = {arXiv:1506.02025v3},
+	volume = "",
+	number = "",
+	pages = "",
+	year = "2015",
+	DOI = "https://doi.org/10.48550/arXiv.1506.02025"
+}
+@article{KBTG2013,
+	author = {Robert R. Kerr and Anthony N. Burkitt  and Doreen A. Thomas and Matthieu Gilson and David B. Grayden},
+	title = {Delay Selection by Spike-Timing-Dependent Plasticity in Recurrent Networks of Spiking Neurons Receiving Oscillatory Inputs},
+	journal = {PLoS Comput. Biol.},
+	volume = "9",
+	number = "",
+	pages = "e1002897",
+	year = "2013",
+	DOI = "https://doi.org/10.1371/journal.pcbi.1002897"
+}
+@article{HKTI2016,
+	author = {Hideyuki Kato and Tohru Ikeguchi},
+	title = {Oscillation, Conduction Delays, and Learning Cooperate to Establish Neural Competition in Recurrent Networks},
+	journal = { PLoS ONE},
+	volume = "11",
+	number = "",
+	pages = "e0146044",
+	year = "2016",
+	DOI = "https://doi.org/10.1371/journal.pone.0146044"
+}
+@article{MAVT2017,
+	author = {Mojtaba Madadi Asl and Alireza Valizadeh and Peter A. Tass },
+	title = {Dendritic and Axonal Propagation Delays Determine Emergent Structures of Neuronal Networks with Plastic Synapses},
+	journal = {Sci. Rep.},
+	volume = "7",
+	number = "",
+	pages = "39682",
+	year = "2017",
+	DOI = "https://doi.org/10.1038/srep39682"
+}
+@article{BSEI2010,
+	author = {Botond Szatmáry and Eugene M. Izhikevich},
+	title = {Spike-Timing Theory of Working Memory},
+	journal = {PLOS Comput. Biol.},
+	volume = "6",
+	number = "",
+	pages = "e1000879",
+	year = "2010",
+	DOI = "https://doi.org/10.1371/journal.pcbi.1000879"
+}
+
+@article{EIAS2018,
+	author = {Akihiro Eguchi and James B. Isbister and Nasir Ahmad and Simon Stringer},
+	title = {The Emergence of Polychronization and Feature Binding in a Spiking Neural Network Model of the Primate Ventral Visual System},
+	journal = {Psychological Review},
+	volume = "125",
+	number = "",
+	pages = "545–571",
+	year = "2018",
+	DOI = "https://doi.org/10.1037/rev0000103"
+}
+
+@article{TM2017,
+	author = {Takashi Matsubara},
+	title = {Conduction Delay Learning Model for Unsupervised and Supervised Classification of Spatio-Temporal Spike Patterns},
+	journal = {Front. Comput. Neurosci.},
+	volume = "11",
+	number = "",
+	pages = "104",
+	year = "2017",
+	DOI = "https://doi.org/10.3389/fncom.2017.00104"
+}
+
+@article{TM2017,
+	author = {Takashi Matsubara},
+	title = {Conduction Delay Learning Model for Unsupervised and Supervised Classification of Spatio-Temporal Spike Patterns},
+	journal = {Front. Comput. Neurosci.},
+	volume = "11",
+	number = "",
+	pages = "104",
+	year = "2017",
+	DOI = "https://doi.org/10.3389/fncom.2017.00104"
+}
+
+@article{HHM2023,
+	author = {Ilyass Hammouamri and Ismail Khalfaoui-Hassani and Timothée Masquelier},
+	title = {Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings},
+	journal = {arXiv},
+	volume = "",
+	number = "",
+	pages = "2306.17670",
+	year = "2023",
+	DOI = "https://doi.org/10.48550/arXiv.2306.17670"
+}
+
+@article{ITT2023,
+	author = {Ismail Khalfaoui-Hassani and Thomas Pellegrini and Timothée Masquelier},
+	title = {Dilated convolution with learnable spacings},
+	journal = {arXiv},
+	volume = "",
+	number = "",
+	pages = "2112.03740v4",
+	year = "2023",
+	DOI = "https://doi.org/10.48550/arXiv.2112.03740"
+}

paper/sections/delays/Delays.md

@@ -14,20 +14,23 @@ This section introduces a simple method to solve the sound localization problem
 
 ### Introduction
 
-Studying the computational properties of axonal transmissions goes as far back as in {cite:p}`Jeffress1948`. In this study, it was shown that with the right network setup, axonal delays can be utilized to transform a temporal cue to a spatial one for sound localization. This study was a leading study in terms of using delays explicitly to explain a neuronal function. It paved the way for others to follow, like the study by {cite:p}`KLWH2001`, where they investigated the question of how ITD computation maps can arise ontogenetically in the laminar nucleus of the barn owl. They showed that interaural time differences (ITD) computational maps emerge from the combined effect of a Hebbian spike-based learning rule and its transmission along the presynaptic axon. Thus, from this study, another role of axonal delays can be inferred. They shape network structure when coupled with temporal learning rules. Based on this insight, several studies investigated the combined effect of spike timing-dependent plasticity (STDP), axonal conduction delays and oscillatory activity on recurrent connections in spiking networks. {cite:p}`KBTG2013` demonstrated the selective potentiation of recurrent connections when the beforementioned computational considerations are taken into account. Also, {cite:p}`HKTI2016` showed that neural selection for memory formation depends on neural competition. In turn, for neural competition to emerge in recurrent networks, spontaneously induced neural oscillation coupled with STDP and axonal delays are a perquisite. 
+:::{Warning}
+Karim here: I have checked the main intro and the intro here and I see no common passages. Thus, I recommend moving all to main intro after the first paragraph. However, if done so, the main intro of delays would be visibly larger than the other sections. In this regard, I leave to you guys to remove any passages you guys deem not necessary as both of you will have the bigger picture. I have made some small edits to improve it, I hope that helps. 
+:::
 
-Coupling conduction delays with STDP seems like a reasonable choice. The sign of the STDP rule depends on the order of post- and pre-synpatic spiking, which axonal delays can effectively reverse. For example, if the presynaptic spikes arrive at the synapse before the backpropagated action potential this would lead a synaptic depression. However, reducing the axonal transmission speed would lead to potentiation. In this line of thought, {cite:p}`MAVT2017` studied the combined role of delays and STDP on the emergent synaptic structure in neural networks. It was shown that, qualitatively different connectivity patterns arise due to the interplay between axonal and dendritic delays, as the synapse and cell body can have different temporal spike order. 
+Studying the computational properties of axonal transmissions goes as far back as in {cite:p}`Jeffress1948`. In this study, it was shown that with the right network setup, axonal delays can be utilized to transform a temporal cue to a spatial one for sound localization. This study was a leading study in terms of using delays explicitly to explain a neuronal function. It paved the way for others to follow, like the study by {cite:p}`KLWH2001`, where they investigated the question of how ITD computation maps can arise ontogenetically in the laminar nucleus of the barn owl. They showed that interaural time differences (ITD) computational maps emerge from the combined effect of a Hebbian spike-based learning rule and its transmission along the presynaptic axon. Thus, from this study, another role of axonal delays can be inferred. They shape network structure when coupled with temporal learning rules. Based on this insight, several studies investigated the combined effect of spike timing-dependent plasticity (STDP), axonal conduction delays and oscillatory activity on recurrent connections in spiking networks. In one of the studies, {cite:p}`KBTG2013` demonstrated the selective potentiation of recurrent connections when the beforementioned computational considerations are taken into account. Also, {cite:p}`HKTI2016` showed that neural selection for memory formation depends on neural competition. In turn, for neural competition to emerge in recurrent networks, spontaneously induced neural oscillation coupled with STDP and axonal delays are a perquisite. 
 
-Aside from their role in modeling cortical functions or shaping a network's synaptic structure, another line of research emerged from the seminal work by {cite:p}`EMI2006`. They showed that when including conduction delays and spike-timing dependent plasticity (STDP) into their simulation of realistic neural models, polychronous groups of neurons emerge. These groups show time-locked spiking pattern with millisecond precision. Subsequent studies investigated the properties and functions of such neuronal groups. For example, {cite:p}`BSEI2010` demonstrated the natural emergence of large memory content and working memory when the neuronal model exploits temporal codes. Specifically, short term plasticity can briefly strengthen the synapses of specific polychronous neuronal groups (PNG) resulting in an enchantment in their spontaneous reactivation rates.  In a qualitatively different study, {cite:p}`EIAS2018` showed that networks that exhibit PNG possess potential capabilities that might solve the dynamic binding problem. These networks respond with stable spatio-temporal spike trains when presented with input images in the form of randomized Poisson spike trains. The functionality of these kind of networks emerged due to the interplay of various factors including: i) random distribution of axonal delays ii) STDP ii) lateral, bottom-up and top-down synaptic connections. 
+From the above, the intuition behind coupling neuronal delays with STDP can be reasoned about. The sign of the STDP rule depends on the order of post- and pre-synpatic spiking, which axonal delays can effectively reverse. For example, the arrival of presynaptic spikes at the synapse before the backpropagated action potential leads to a synaptic depression. However, reducing the axonal transmission speed would lead to potentiation. In this line of thought, {cite:p}`MAVT2017` studied the combined role of delays and STDP on the emergent synaptic structure in neural networks. It was shown that, qualitatively different connectivity patterns arise due to the interplay between axonal and dendritic delays, as the synapse and cell body can have different temporal spike order. 
 
-Finally, it should be noted that most of the studies that incorporate axonal and/or dendritic delays, include them as a non-learnable parameter. Few studies investigated the possibility of training transmission delays in order to enhance the computational capabilities of spiking neural networks (SNN). {cite:p}`TM2017` proposed an algorithm that modifies the axonal delays and synaptic efficacy in both supervised and unsupervised approaches.  The learning method used approximates the Expectation-Maximization (EM) algorithm and after training, the network learns to  map spatio-temporal input-output spike patterns. Thus, EM is one way to train SNN that are cast as probabilistic models. Another approach that exploits the massive infrastructure that is laid out the deep learning literature is the work by {cite:p}`hammouamri2024learning`. In this work, delays are represented as 1D convolutions through time, where the kernels include a single per-synapse non-zero weight. The temporal position of these non-zero weights corresponds to the desired delays. The proposed method co-trains weights and delays and is based on the Dilated Convolution
-with Learnable Spacings (DCLS) algorithm [{cite:p}`ITT2023`].  
+Aside from their role in modeling cortical functions or shaping a network's synaptic structure, another line of research emerged from the seminal work by {cite:p}`EMI2006`. They showed that when including conduction delays and spike-timing dependent plasticity (STDP) into their simulation of realistic neural models, polychronous groups of neurons emerge. These groups show time-locked spiking pattern with millisecond precision. Subsequent studies investigated the properties and functions of such neuronal groups. For example, {cite:p}`BSEI2010` demonstrated the natural emergence of large memory content and working memory when the neuronal model exploits temporal codes. Specifically, short term plasticity can briefly strengthen the synapses of specific polychronous neuronal groups (PNG) resulting in an enchantment in their spontaneous reactivation rates.  In a qualitatively different study, {cite:p}`EIAS2018` showed that networks that exhibit PNG possess potential capabilities that might solve the dynamic binding problem. These networks respond with stable spatio-temporal spike trains when presented with input images in the form of randomized Poisson spike trains. The functionality of these kind of networks emerged due to the interplay of various factors including: i) random distribution of axonal delays ,ii) STDP and iii) lateral, bottom-up and top-down synaptic connections. 
+
+Finally, it should be noted that most of the studies that incorporate axonal and/or dendritic delays, include them as a non-learnable parameter. Few studies investigated the possibility of training transmission delays in order to enhance the computational capabilities of spiking neural networks (SNN). {cite:p}`TM2017` proposed an algorithm that modifies the axonal delays and synaptic efficacy in both supervised and unsupervised approaches.  The learning method involbed used approximates of the Expectation-Maximization (EM) algorithm and after training, the network learns to map spatio-temporal input-output spike patterns. Thus, EM is one way to train SNNs that are cast as probabilistic models. Another approach that exploits the massive infrastructure that is laid out the deep learning literature is the work by {cite:p}`hammouamri2024learning`. In this work, delays are represented as 1D convolutions through time, where the kernels include a single per-synapse non-zero weight. The temporal position of these non-zero weights corresponds to the desired delays. The proposed method co-trains weights and delays and is based on the Dilated Convolution with Learnable Spacings (DCLS) algorithm ({cite:p}`ITT2023`).  
 
 In this work we propose a delay learning algorithm that is simple and efficient. The delay learning is mediated by a differentiable delay layer (DDL). This layer can be inserted between any two layers in an SNN in order to learn the appropriate delay to solve a machine learning task. This DDL is architecture agnostic. Also, the method is designed to learn delays separately from synaptic weights. 
 
 ### Methods
 
-The DDL is, mainly, based on a 1D version of the spatial transformer (STN) network {cite:p}`JSZK2015`. The STN is a differentiable module that can be added into convolutional neural networks (CNNs) architectures to empower them with the ability to spatially transform feature maps in a differentiable way. This addition leads to CNNs models that are invariant to various spatial transformations like translation, scaling and rotation. Image manipulations are inherently  not differentiable, because pixels are a discrete. However, this problem is overcome by the application of an interpolation  (for example bi-linear) after the spatial transformation. 
+The DDL is, mainly, based on a 1D version of the spatial transformer (STN) network {cite:p}`JSZK2015`. The STN is a differentiable module that can be added into convolutional neural networks (CNNs) architectures to empower them with the ability to spatially transform feature maps in a differentiable way. This addition leads to CNN models that are invariant to various spatial transformations like translation, scaling and rotation. Image manipulations are inherently  not differentiable, because pixels are a discrete. However, this problem is overcome by the application of an interpolation  (for example bi-linear) after the spatial transformation. 
 
 The DDL is a 1D version of the spatial transformer where the only transformation done is translation. Translation of a spike along the time dimension can be thought of as a translation of a pixel along the spatial coordinates. The general affine transformation matrix for the 2D case takes the form in the following equation: