Always ravenous for information, the brain acts like a forager that never knows where its next meal is coming from. So it gorges when it finds digestible data. Witness the way snacking on visual hors d’oeuvres helps your brain process speech.

Experiment 1  Jot down which letter you think is mouthed in each of the 10 pictures at right (no fair peeking at the second set of pictures below or at the answers at the bottom of the page).

1

2

3

4

5

678910

You should easily identify at least one letter: O. If you don’t know precisely which letter is mouthed, guess whether it is a vowel or a consonant. Hint: Mouth 4 is clearly a vowel and mouth 2 a consonant. You can make these reasonable guesses because your brain has learned how to gobble up visual information relevant to speech, even when it’s consuming more than enough acoustic information to understand spoken words.

Experiment 2  To see just how useful redundant visual information can be, watch a TV program with the volume turned so low that you can barely hear what the actors are saying. Then close your eyes, noticing how you miss words without the visual information.

Experiment 3  In addition to using visual information, your brain uses context to understand speech. The 10 mouths from the first experiment are sorted into three groups, below. Guess which mouths correspond to the letters next to each group (the answers are at the bottom of the page). Your performance should improve considerably over that for Experiment 1.

Experiment 4  Stand directly behind a partner, facing a wall mirror. Fix your gaze on the reflected image of your partner’s mouth. Then have your partner mouth—but not say aloud—gah while simultaneously you softly utter bah. If you synchronize your quiet bah with your partner’s silent gah (counting down from three will help), both of you may very well hear dah. This illusion occurs because your brain relies so much on vision that your eyes influence your ears. Harry McGurk, the late British psychologist who discovered the effect, theorized that when the brain is confronted with conflicting acoustic and visual information, it interpolates (ergo dah from bah and gah). Test the theory by standing in front of your friend. Now mouth gah into the mirror while your friend says bah. Repeat the procedure with your eyes closed and hear the dah turn back into a bah.

Computer scientists are beginning to mimic the brain’s strategy of combining visual and acoustic information by feeding live video images of speakers’ mouths into voice-recognition software. It won’t be long before you can conduct a decent conversation with your laptop. But watch out: The more that computer becomes like your brain, the more voracious its sensory appetite will be.

A,E,I,O,U

J,K,LC,T

 Illustration by Bill Russell

Answers: Experiment 1, (1) C; (2) L; (3) E; (4) I; (5) T; (6) U; (7) K; (8) O; (9) A; (10) J. Experiment 3, (1) O; (2) E; (3) U; (4) A; (5) I; (6) L; (7) K; (8) J; (9) T; (10) C.