Neuroscience and Computational Biology

Auditory cortex signal processing; brain development and plasticity; sensorimotor integration; neuromechanical systems; speech recognition; neuromorphic sensors, control and VLSI; computational neuroscience; cell-based sensors

ISR is a longtime leader in advancing understanding of neural processing in the brain's auditory system, including speech processing and sound localization. Our faculty and students have made neuroscience-based advances in signal processing principles and solutions, and have developed novel neuromorphic architectures for intelligent systems. We are active in NIH BRAIN Initiative research, using neural modeling to establish causal links between neural activity and behavior. We also developed multi-pitch tracking for adverse environments, a communication technology that pulls speech out of noise and can radically improve sound quality over cell phones and in hearing aids. ISR researchers have been key in the establishment of the university’s Brain and Behavior Initiative.

Recent news

Recent publications

2020

A Deep 2-Dimensional Dynamical Spiking Neuronal Network for Temporal Encoding trained with STDP

Matthew Evanusa, Cornelia Fermüller, Yiannis Aloimonos

The researchers show that a large, deep layered spiking neural network with dynamical, chaotic activity mimicking the mammalian cortex with biologically-inspired learning rules, such as STDP, is capable of encoding information from temporal data.

arXiv.org

Hybrid Backpropagation Parallel Reservoir Networks

Matthew Evanusa, Snehesh Shrestha, Michelle Girvan, Cornelia Fermüller, Yiannis Aloimonos

Demonstrates the use of a backpropagation hybrid mechanism for parallel reservoir computingwith a meta ring structure and its application on a real-world gesture recognition dataset. This mechanism can be used as an alternative to state of the art recurrent neural networks, LSTMs and GRUs.

arXiv.org

2022

NLGC: Network Localized Granger Causality with Application to MEG Directional Functional Connectivity Analysis

Behrad Soleimani, Proloy Das, I.M. Dushyanthi Karunathilake, Stefanie E. Kuchinsky, Jonathan Z. Simon, Behtash Babadi

Introduces the Network Localized Granger Causality (NLGC) inference paradigm, which models the source dynamics as latent sparse multivariate autoregressive processes and estimates their parameters directly from the MEG measurements, integrated with source localization, and employs the resulting parameter estimates to produce a precise statistical characterization of the detected GC links.

bioRxiv.org

2021

Direct Extraction of Signal and Noise Correlations from Two-Photon Calcium Imaging of Ensemble Neuronal Activity

Anuththara Rupasinghe, Nikolas Francis, Ji Liu, Zac Bowen, Patrick Kanold, Behtash Babadi

Proposes a methodology to directly estimate both signal and noise correlations from two-photon imaging observations, without requiring an intermediate step of spike deconvolution.

eLife

2020

Dynamic estimation of auditory temporal response functions via state-space models with Gaussian mixture process noise

Sina Miran, Behtash Babadi, Alessandro Presacco, Jonathan Simon, Michael Fu, Steven Marcus

This research develops efficient algorithms for inferring the parameters of a general class of Gaussian mixture process noise models from noisy and limited observations, and utilize them in extracting the neural dynamics that underlie auditory processing from magnetoencephalography (MEG) data in a cocktail party setting.

PLOS Computational Biology

Multitaper Analysis of Semi-Stationary Spectra from Multivariate Neuronal Spiking Observations

Anuththara Rupasinghe, Behtash Babadi

Extracting the spectral representations of neural processes that underlie spiking activity is key to understanding how brain rhythms mediate cognitive functions. This work develops a multitaper spectral estimation methodology that can be directly applied to multivariate spiking observations to extract the semi-stationary spectral density of the latent non-stationary processes that govern spiking activity.

IEEE Transactions on Signal Processing

Robust inference of neuronal correlations from blurred and noisy spiking observations

Anuththara Rupasinghe, Behtash Babadi

Proposes an algorithm to directly estimate neuronal correlations from ensemble two-photon imaging data, by integrating techniques from point process modeling and variational Bayesian inference, with no recourse to intermediate spike deconvolution.

2020 54th Annual Conference on Information Sciences and Systems

Neuro-current response functions: A unified approach to MEG source analysis under the continuous stimuli paradigm

Proloy Das, Christian Brodbeck, Jonathan Z. Simon, Behtash Babadi

A principled modeling and estimation paradigm for MEG source analysis tailored to extracting the cortical origin of electrophysiological responses to continuous stimuli.

NeuroImage

Granger Causal Inference from Indirect Low-Dimensional Measurements with Application to MEG Functional Connectivity Analysis

Behrad Soleimani, Proloy Das, Joshua Kulasingham, Jonathan Z. Simon, Behtash Babadi

The authors consider the problem of determining Granger causal influences among sources that are indirectly observed through low-dimensional and noisy linear projections. They model the source dynamics as sparse vector autoregressive processes and estimate the model parameters directly from the observations, with no recourse to intermediate source localization.

54th Conference on Information Sciences and Systems

2021

A potential role for binocular integration without binocular disparity information in primary visual cortex

Ethan Cheng, Daniel Butts

How information from both eyes aligns in the primary visual cortex (V1) depends on the vergence of the eyes and the depth of a given object, resulting in a particular binocular disparity. However, a significant fraction of V1 neurons are binocular but not modulated by binocular disparity, raising the question of how their outputs can be meaningful.

Journal of Vision

Decision-related feedback in visual cortex lacks spatial selectivity

Katrina Quinn, Lenka Seillier, Daniel Butts, Hendrikje Nienborg

Feedback in the brain is thought to convey contextual information that underlies our flexibility to perform different tasks. The authors show that although the behavior is spatially selective, using only task-relevant information, modulation by decision-related feedback is spatially unselective.

Nature Communications

2020

A latent variable approach to decoding neural population activity

Matthew Whiteway, Bruno Averbeck, Daniel Butts

The authors propose a new decoding framework that exploits the low-dimensional structure of neural population variability by removing correlated variability unrelated to the decoded variable, then decoding the resulting denoised activity.

biorXiv.org

2019

The quest for interpretable models of neural population activity

Matthew Whiteway, Daniel Butts

Research explores latent variable models for neural recording coordinated activity of large neuron populations in brain function.

Current Opinion in Neurobiology

2020

A Deep 2-Dimensional Dynamical Spiking Neuronal Network for Temporal Encoding trained with STDP

Matthew Evanusa, Cornelia Fermüller, Yiannis Aloimonos

The researchers show that a large, deep layered spiking neural network with dynamical, chaotic activity mimicking the mammalian cortex with biologically-inspired learning rules, such as STDP, is capable of encoding information from temporal data.

arXiv.org

Hybrid Backpropagation Parallel Reservoir Networks

Matthew Evanusa, Snehesh Shrestha, Michelle Girvan, Cornelia Fermüller, Yiannis Aloimonos

Demonstrates the use of a backpropagation hybrid mechanism for parallel reservoir computingwith a meta ring structure and its application on a real-world gesture recognition dataset. This mechanism can be used as an alternative to state of the art recurrent neural networks, LSTMs and GRUs.

arXiv.org

2022

Individual differences in speech stream segregation and working memory differentially predict performance on a stochastic figure-ground task

Regina Calloway, Michael Johns, Ian Phillips, Valerie Karuzis, Kelsey Dutta, Ed Smith, Shihab Shamma, Matthew Goupell, Stefanie Kuchinsky

Understanding speech in noisy environments can be challenging and requires listeners to accurately segregate a target speaker from irrelevant background noise. In this research, an online SFG task with complex stimuli consisting of a sequence of inharmonic pure-tone chords was administered to 37 young, normal hearing adults, to have a more pure measure of auditory stream segregation that does not rely on linguistic stimuli. Detection of target figure chords consisting of 4, 6, 8, or 10 temporally coherent tones amongst a background of randomly varying tones was measured. Increased temporal coherence (i.e., number of tones in a figure chord) resulted in higher accuracy and faster reaction times (RTs). At higher coherence levels, faster RTs were associated with better scores on a standardized speech-in-noise recognition task. Increased working memory capacity hindered SFG accuracy as the tasked progressed, whereas self-reported musicianship modulated the relationship between speech-in-noise recognition and SFG accuracy. Overall, results demonstrate that the SFG task could serve as an assessment of auditory stream segregation that is sensitive to capture individual differences in working memory capacity and musicianship.

arXiv.org

2021

The Mirrornet: Learning Audio Synthesizer Controls Inspired by Sensorimotor Interaction

Yashish M. Siriwardena, Guilhem Marion, Shihab Shamma

The recently proposed “MirrorNet,” a constrained autoencoder architecture. In this paper, the MirrorNet is applied to learn, in an unsupervised manner, the controls of a specific audio synthesizer (DIVA) to produce melodies only from their auditory spectrograms. The results demonstrate how the MirrorNet discovers the synthesizer parameters to generate the melodies that closely resemble the original and those of unseen melodies, and even determine the best set parameters to approximate renditions of complex piano melodies generated by a different synthesizer. This generalizability of the MirrorNet illustrates its potential to discover from sensory data the controls of arbitrary motor-plants such as autonomous vehicles.

arXiv.org

The Music of Silence: Part I: Responses to Musical Imagery Encode Melodic Expectations and Acoustics

Guilhem Marion, Giovanni M. Di Liberto, Shihab Shamma

It is well known that the human brain is activated during musical imagery: the act of voluntarily hearing music in our mind without external stimulation. It is unclear, however, what the temporal dynamics of this activation are, as well as what musical features are precisely encoded in the neural signals. This study uses an experimental paradigm with high temporal precision to record and analyze the cortical activity during musical imagery. This study reveals that neural signals encode music acoustics and melodic expectations during both listening and imagery. Crucially, it is also found that a simple mapping based on a time-shift and a polarity inversion could robustly describe the relationship between listening and imagery signals.

Journal of Neuroscience

The Music of Silence: Part II: Music Listening Induces Imagery Responses

Giovanni M. Di Liberto, Guilhem Marion, Shihab Shamma

Music perception depends on our ability to learn and detect melodic structures. It has been suggested that our brain does so by actively predicting upcoming music notes, a process inducing instantaneous neural responses as the music confronts these expectations. Here, we studied this prediction process using EEGs recorded while participants listen to and imagine Bach melodies. Specifically, we examined neural signals during the ubiquitous musical pauses (or silent intervals) in a music stream and analyzed them in contrast to the imagery responses. We find that imagined predictive responses are routinely co-opted during ongoing music listening. These conclusions are revealed by a new paradigm using listening and imagery of naturalistic melodies.

Journal of Neuroscience

2020

Learning Speech Production and Perception through Sensorimotor Interactions

Shihab Shamma, Prachi Patel, Shoutik Mukherjee, Guilhem Marion, Bahar Khalighinejad, Cong Han, Jose Herrero, Stephan Bickel, Ashesh Mehta, Nima Mesgarani

Action and perception are closely linked in many behaviors necessitating a close coordination between sensory and motor neural processes so as to achieve a well-integrated smoothly evolving task performance. To investigate the detailed nature of these sensorimotor interactions, and their role in learning and executing the skilled motor task of speaking, the authors analyzed ECoG recordings of responses in the high-γ band (70–150 Hz) in human subjects while they listened to, spoke, or silently articulated speech. They found elaborate spectrotemporally modulated neural activity projecting in both “forward” (motor-to-sensory) and “inverse” directions between the higher-auditory and motor cortical regions engaged during speaking. Furthermore, mathematical simulations demonstrate a key role for the forward projection in “learning” to control the vocal tract, beyond its commonly postulated predictive role during execution. These results therefore offer a broader view of the functional role of the ubiquitous forward projection as an important ingredient in learning, rather than just control, of skilled sensorimotor tasks.

Cerebral Cortex Communications (Oxford Academics)

2022

NLGC: Network Localized Granger Causality with Application to MEG Directional Functional Connectivity Analysis

Behrad Soleimani, Proloy Das, I.M. Dushyanthi Karunathilake, Stefanie E. Kuchinsky, Jonathan Z. Simon, Behtash Babadi

Introduces the Network Localized Granger Causality (NLGC) inference paradigm, which models the source dynamics as latent sparse multivariate autoregressive processes and estimates their parameters directly from the MEG measurements, integrated with source localization, and employs the resulting parameter estimates to produce a precise statistical characterization of the detected GC links.

bioRxiv.org

Algorithms for Estimating Time-Locked Neural Response Components in Cortical Processing of Continuous Speech

Joshua Kulasingham, Jonathan Simon

The Temporal Response Function (TRF) is a linear model of neural activity time-locked to continuous stimuli, including continuous speech. TRFs based on speech envelopes typically have distinct components that have provided remarkable insights into the cortical processing of speech. However, current methods may lead to less than reliable estimates of single-subject TRF components. The authors compare two established methods, in TRF component estimation, and also propose novel estimation algorithms that utilize prior knowledge of these components, bypassing the full TRF estimation.

bioRxiv.org

2021

Cortical tracking of voice pitch in the presence of multiple speakers depends on selective attention

Christian Brodbeck, Jonathan Simon

Voice pitch carries linguistic as well as non-linguistic information. Previous studies have described cortical tracking of voice pitch in clean speech, with responses reflecting both pitch strength and pitch value. However, pitch is also a powerful cue for auditory stream segregation, especially when competing streams have pitch differing in fundamental frequency, as is the case when multiple speakers talk simultaneously. The authors investigate how cortical speech pitch tracking is affected in the presence of a second, task-irrelevant speaker.

bioRxiv.org

Neural markers of speech comprehension: measuring EEG tracking of linguistic speech representations, controlling the speech acoustics

Marlies Gillis, Jonas Vanthornhout, Jonathan Simon, Tom Francart and Christian Brodbeck

The authors evaluate the potential of several recently proposed linguistic representations as neural markers of speech comprehension.

Journal of Neuroscience

Cortical Processing of Arithmetic and Simple Sentences in an Auditory Attention Task

Joshua Kulasingham, Neha Joshi, Mohsen Rezaeizadeh, Jonathan Simon

A study that shows the neural responses to arithmetic and language are especially well segregated during the cocktail party paradigm. A correlation with behavior suggests that they may be linked to successful comprehension or calculation.

Journal of Neuroscience

Local versus long-range connectivity patterns of auditory disturbance in schizophrenia

Stephanie Hare, Bhim Adhikari, Xiaoming Du, Laura Garcia, Heather Bruce, Peter Kochunov, Jonathan Simon, Elliot Hong

The study investigates how local and long-range functional connectivity is associated with auditory perceptual disturbances (APD) in schizophrenia.

Schizophrenia Research

Cortical Processing of Arithmetic and Simple Sentences in an Auditory Attention Task

Joshua Kulasingham, Neha Joshi1, Mohsen Rezaeizadeh, Jonathan Simon

Cortical processing of arithmetic and of language rely on both shared and task-specific neural mechanisms, which should also be dissociable from the particular sensory modality used to probe them. Here, spoken arithmetical and non-mathematical statements were employed to investigate neural processing of arithmetic, compared to general language processing, in an attention-modulated cocktail party paradigm.

biorxiv.org

2020

Poststroke acute dysexecutive syndrome, a disorder resulting from minor stroke due to disruption of network dynamics

Elisabeth Marsh, Christian Brodbeck, Rafael Llinas, Dania Mallick, Joshua Kulasingham, Jonathan Simon, and Rodolfo Llinas

For the first time measurable physical evidence is provided of diminished neural processing within the brain after a stroke. It suggests that poststroke acute dysexecutive syndrome (PSADES) is the result of a global connectivity dysfunction.

Proceedings of the National Academy of Sciences of the United States of America

Neural speech restoration at the cocktail party: Auditory cortex recovers masked speech of both attended and ignored speakers

Christian Brodbeck, Alex Jiao, L. Elliot Hong, Jonathan Simon

Humans are remarkably skilled at listening to one speaker out of an acoustic mixture of several speech sources. Two speakers are easily segregated, even without binaural cues, but the neural mechanisms underlying this ability are not well understood. One possibility is that early cortical processing performs a spectrotemporal decomposition of the acoustic mixture, allowing the attended speech to be reconstructed via optimally weighted recombinations that discount spectrotemporal regions where sources heavily overlap. Using human magnetoencephalography (MEG) responses to a 2-talker mixture, the authors show evidence for an alternative possibility, in which early, active segregation occurs even for strongly spectrotemporally overlapping regions.

PLOS Biology

High Gamma Cortical Processing of Continuous Speech in Younger and Older Listeners

Peng Zan, Alessandro Presacco, Samira Anderson, Jonathan Simon

Aging is associated with an exaggerated representation of the speech envelope in auditory cortex. The relationship between this age-related exaggerated response and a listener’s ability to understand speech in noise remains an open question. Here, information-theory-based analysis methods are applied to magnetoencephalography recordings of human listeners, investigating their cortical responses to continuous speech, using the novel nonlinear measure of phase-locked mutual information between the speech stimuli and cortical responses. The cortex of older listeners shows an exaggerated level of mutual information, compared with younger listeners, for both attended and unattended speakers.

Journal of Neurophysiology

Exaggerated cortical representation of speech in older listeners: mutualinformation analysis

Peng Zan, Alessandro Presacco, Samira Anderson, Jonathan Simon

Information-theory-based analysis methods are applied to magnetoencephalography recordings of human listeners, investigating their cortical responses to continuous speech, using the novel nonlinear measure of phase-locked mutual information between the speech stimuli and cortical responses. This information-theory-based analysis provides new, and less coarse-grained, results regarding age-related change in auditory cortical speech processing, and its correlation with cognitive measures, com-pared with related linear measures.

Journal of Neurophysiology

Continuous Speech Processing

Christian Brodbeck, Jonathan Simon

Speech processing in the human brain is grounded in non-specific auditory processing in the general mammalian brain, but relies on human-specific adaptations for processing speech and language. For this reason, many recent neurophysiological investigations of speech processing have turned to the human brain, with an emphasis on continuous speech. This article considers the substantial progress that has been made using the phenomenon of 'neural speech tracking,' in which neurophysiological responses time-lock to the rhythm of auditory (and other) features in continuous speech.

Current Opinion in Physiology

Dynamic estimation of auditory temporal response functions via state-space models with Gaussian mixture process noise

Sina Miran, Behtash Babadi, Alessandro Presacco, Jonathan Simon, Michael Fu, Steven Marcus

This research develops efficient algorithms for inferring the parameters of a general class of Gaussian mixture process noise models from noisy and limited observations, and utilize them in extracting the neural dynamics that underlie auditory processing from magnetoencephalography (MEG) data in a cocktail party setting.

PLOS Computational Biology

Neuro-current response functions: A unified approach to MEG source analysis under the continuous stimuli paradigm

Proloy Das, Christian Brodbeck, Jonathan Z. Simon, Behtash Babadi

A principled modeling and estimation paradigm for MEG source analysis tailored to extracting the cortical origin of electrophysiological responses to continuous stimuli.

NeuroImage

Granger Causal Inference from Indirect Low-Dimensional Measurements with Application to MEG Functional Connectivity Analysis

Behrad Soleimani, Proloy Das, Joshua Kulasingham, Jonathan Z. Simon, Behtash Babadi

The authors consider the problem of determining Granger causal influences among sources that are indirectly observed through low-dimensional and noisy linear projections. They model the source dynamics as sparse vector autoregressive processes and estimate the model parameters directly from the observations, with no recourse to intermediate source localization.

54th Conference on Information Sciences and Systems


Top