Temporal Correlation and Its Role in Multisensory Feature Integration and Binding

Abstract

Our successful interaction with the environment requires the brain to appropriately combine and segregate sensory information coming from various events. These events frequently produce energy and signals that are processed by different sensory systems. These signals, such as the voice and the mouth movements of a speaker, are often temporally correlated (i.e., they change together over time). Human observers can use these correlation cues to facilitate feature binding—which results in the formation of a multisensory perceptual object. This binding of signals across sensory modalities often confers a constellation of behavioral and perceptual benefits. For example, the intelligibility of speech in noise is improved when we can see the face of the person speaking. Because complex sensory environments can generate weak—but significant—correlations between sensory signals from separate events, evaluating the strength of correlation would enable the selection of the appropriate signals to be bound. We investigated how observers utilized different levels of temporal correlation during multisensory behaviors. We found that detection of amplitude modulation (AM) in auditory and visual stimuli improved in a linear manner with the temporal correlation of those AM features. The same benefit conferred by AM correlation was also observed in the detection of an orthogonal frequency modulation feature. We also discovered a novel phenomenon where participant behaviors were uniquely phase-shifted relative to stimulus correlation. We hypothesized that this transformation reflected individual differences in the temporal processing abilities of participants’ auditory and visual systems. Phase shifts were accounted for by simulating a lag in either the auditory or visual AM and separate experiments show that phase shifts across participants were related to their differences in auditory and visual reaction time. These experiments indicate that the brain’s computation of correlation is more sophisticated than previously shown and suggest a potential mechanism for the binding of cues in complex sensory landscapes