Today a new generation of scientists is discovering just how powerful the McGurk Effect really is. University of California, Los Angeles, psychologist Ladan Shams has been able to create a McGurk-like illusion by displaying a flash paired with varying numbers of corresponding beeps. If Shams delivered a flash with a pair of beeps, participants were more likely to say they also saw two flashes.

These illusions provide evidence of a powerful strategy our brains 
use to cope with the uncertain signals we get from our senses. 
Certain regions of the brain take input from two or more senses and then combine them in a single sensory channel to sharpen the information overall.

Michael Beauchamp and Audrey Nath, neuroscientists at the University of Texas Health Science Center at Houston, have pinpointed one of the crucial nodes where these streams of information meet. They delivered short magnetic pulses that briefly shut down different parts of the brain. When they applied the pulses to a strip of the brain near the ear, a region called the superior temporal sulcus, the McGurk Effect was diminished.

Beauchamp and Nath followed up on that study with a new one in which they scanned people’s brains with functional magnetic resonance imaging (fMRI) as they played McGurk videos of mismatched sounds and lip movements. They found that the left superior temporal sulcus became more active in people who experienced the illusion and remained less active in those who didn’t. Beauchamp and Nath’s work suggests that when the illusion occurs, it is because the superior temporal sulcus discounts some of the signals coming from one sensory region of the brain in favor of others.

We don’t mix up our senses willy-nilly, however. There is a window of less than a tenth of a second in which a stimulus from one sense can affect the others. As my misadventure with Netflix showed, my brain was accustomed to balancing sight and sound to make sense of what people were saying without my even noticing, but the sound lag during that episode of Law and Order was so wide that the two sensory streams created confusion instead.

Sight and sound are not the only senses we mingle in our brains. What we touch can affect what we see or hear. Our very understanding of the shape of our own body can be informed not just directly, through our eyes, but also by the pressure of our feet on the ground, the stretch of ligaments in our shoulders, and the wiggle of balance-sensing nerve hairs in our inner ears. Together, our senses let us control our bodies, keeping us from falling over every time we stand up.

But this much integration comes with an astonishing ability to be duped. In 1998 Matthew Botvinick and Jonathan Cohen, two psychologists then at the University of Pittsburgh, found they could make people feel as if a rubber hand were really their own. All they had to do was put a rubber hand in front of their subjects and have them put their real hand behind a screen. The scientists simultaneously began to stroke the real hand and the fake one with paintbrushes. In a matter of seconds, people reported that the rubber hand felt as if it were part of their own body and that they even felt it being stroked.

Neuroscientist Valeria Petkova of the Karolinska Institute in Sweden expanded the rubber hand illusion in 2008. Instead of making people feel a rubber hand was part of their body, she wanted to swap entire bodies. First she had volunteers put on goggles fitted with video screens for lenses. The lenses, in turn, transmitted video from cameras positioned to correspond to a mannequin’s eyes. The cameras were pointed down at the mannequin’s body, so that the mannequin seemed to occupy the precise position of the volunteer’s own body. One scientist stroked the abdomen of the mannequin, while another stroked the abdomens of the volunteers. The effect has been described as a total body swap—the synchronized acts of stroking made participants feel as if the mannequin’s body were their own.

These results were intriguing but based largely on what the volunteers told Petkova about how they felt. That left the researchers wondering what was actually going on in their brains. To find out, they redesigned the experiment so that the volunteers lay in an fMRI scanner that recorded their brain activity. While the scan was in progress, participants looked down at their bodies and a scientist stroked their hand or abdomen. Then the scientists again took participants through the whole-body swap. Once the volunteers felt that the mannequin’s body was their own, the same brain regions became active as when they actually looked down at their own bodies.

The tricks we use to integrate our senses take time to develop. As children grow up, they get better and better at combining sights and sounds. When scientists compare children of the same age, they discover a fascinating pattern: The ones who are better at combining sights and sounds tend to score higher on intelligence tests. It’s possible, some scientists suggest, that helping children combine their senses through training exercises will enable them to do better in school. Manipulating our senses could also help people who have lost a limb. Researchers at the Rehabilitation Institute of Chicago are using the rubber hand illusion to teach amputees to feel artificial limbs as their own.

Learning how the brain mingles its senses can do more than shed light on the latest glitch in streaming video. It may also be able to help some people rewire their sense of reality for their own good.

Carl Zimmer is an award-winning biology writer and author of The Tangled Bank: An 
Introduction to Evolution. He also has a blog called The Loom on this site.