Ladan Shams, a Caltech postdoctoral researcher, and Shinsuke Shimojo, a professor of computation and neural systems at Caltech report that visual signals are influenced significantly by sounds at early cortical levels that have been believed to be "vision specific."
The team's initial behavioral finding was that when an observer is shown one flash of light accompanied by two beeps, the visual system is tricked so that the observer sees two flashes instead of one. In the new study, 13 healthy volunteers were asked to observe the stimuli on a computer screen and judge the number of flashes they saw on the screen.
While the participants performed the task, their brains' electric potentials were recorded from three electrodes positioned in the back of the scalp, where the early visual areas are located.
The researchers found that when the participants perceived the illusion—in other words, when sound changed the visual perception—the activity in the visual areas was modified. Furthermore, the change in activity was similar to that induced by an additional physical flash.
This suggests that the second flash, which is nothing but an illusion and is not due to a visual stimulus but rather caused by sound, invokes activity in the visual areas very similar to that which would be caused by a physical second flash. In short, sound induces a similar effect in this area of the brain to a visual stimulus.
The goal of this study was to get an understanding of how this alteration of vision by sound occurs in the brain. More specifically, the researchers asked whether the change in visual perception is caused by a change in the higher-level areas of the brain that are known to combine information from multiple senses, or whether it is a change that directly affects the activity of the areas that are believed to be exclusively involved in processing visual information.
The main result of the study was that the early visual cortical responses were modulated by accompanied sounds under conditions where the observers experienced the double-flash illusion. This suggests that the activity of the "visual" areas in the brain is affected by sound.
These findings challenge two traditional perspectives on how the brain processes sensory information. The first assumption is that humans are visual animals; vision is the dominant modality and hence not malleable by information from other modalities. Another general belief is that the information from different modalities is processed in the brain in parallel and separate paths.
The findings show that the visual information is affected by the auditory signals while being processed in the "modality-specific" visual pathway. These findings, together with earlier results in other modalities, suggest a paradigm of sensory processing that is more intertwined than segregated.
"The findings have an important implication for the new studies of human perception," says Shimojo. The overwhelming majority of studies in the field of perception have concerned themselves with one modality alone (based on the assumption of modality segregation).
This study, together with other studies indicating vigorous early plasticity and interactions across sensory modalities, is also very encouraging for applications such as sensory aids for children suffering from blindness or dyslexia, for educational applications, for man-made interfaces, and for media and information technology.
A report on this study will appear in the December issue of the journal NeuroReport. Ladan Shams is lead author of the paper.
Contact: Robert Tindol (626) 395-3631
