Chapter
Six
The unity
because the senses are so different. For example, while vision depends largely on
spatial analysis, hearing uses temporal analysis. How are these two very different
processes integrated?
Think of the way you turn your head and eyes to look straight at someone who
calls your name, or the way that the smell and touch and sight of the sandwich
in your hand all seem to belong to the same object. Or think of a cat out hunting.
It listens to the rustling in the undergrowth, creeps carefully between the leaves,
feeling its way with its whiskers, then spies the poor vole, and pounces. Cats and
rats construct a spatial map around themselves in which information from their
eyes, ears, whiskers, and paws is all integrated. When information from both sight
and sound comes from the same position, then responses are enhanced, and
information from one sense can affect responses to another sense in many ways
(Stein, Wallace, and Stanford, 2001).
Somehow decisions are made about what to integrate and when. Principles of
time, space, and inverse effectiveness have been proposed as guiding the inte-
gration process: if cross-modal stimuli arise at the same time and place, and if in
isolation these stimuli evoke relatively weak responses, multisensory integration
is more likely, and likely to be stronger. A more general framework for under-
standing the likelihood of different levels of segregation and integration in mul-
tisensory perception is the Bayesian model, in which new evidence is combined
with prior belief to assess probability (Beierholm, Quartz, and Shams, 2009).
Within the brain, integration depends on multisensory neurons that respond to
input from more than one sense. In the superior colliculus in the midbrain, cells
may respond to more than one sense even at birth, but their multisensory capac-
ity increases with experience and with increasing connections into the cortex.
Integration can give rise to illusions. Ventriloquism (Recanzone, 2009) works
because when you hear a voice speaking words that coincide with movements
of a toy’s mouth, you hear the sounds as though they come from the toy. Exper-
iments with the McGurk effect entail watching a person speaking one phoneme
while listening to a different one, and the result can be a different phoneme alto-
gether. For example, if you are shown someone saying ‘ga’ and played the sound
‘ba’, you will hear ‘da’.
Nonetheless, for most of us, most of the time, the senses remain easily dis-
tinguishable. That is, we are not confused as to whether we heard, saw, or
touched something. This ability is not as obvious as it may seem, because all
the senses work by using the same kinds of neural impulses. So it seems that
some explanation is needed for why they are experienced as so distinct (O’Re-
gan and Noë, 2001). Perhaps we can learn something from the phenomenon
of synaesthesia, in which the senses are not so distinct (see Concept 6.1). Many
people can remember that as children they sometimes heard smells or tasted
sounds, and in some people this mixing of the senses remains part of their
lifelong experience.
Some argue that the more we learn about interaction between senses, the
harder it is to define what a ‘sensory modality’ means: do pain, vestibu-
lar awareness, or thermal perception count as separate modalities, and if
not, why not? And what about awareness of speech or music compared to
general sound perception, or the interactions between apparently distinct