A wiry man in his late 30s hangs near the top of a 40-foot wall, the toes of his climbing shoes barely gripping the sheer face. He clings with his left hand and pauses, swinging his right arm to the side, loosening the muscles. Then he raises his head and lunges, thrusting his right hand above him and catching a rock hold with his fingertips. With a few more moves he arrives at the top. Under normal circumstances this extreme rock climber would be worth watching. But what makes his effort even more remarkable is that he happens to be blind. Born with retinoschisis, a rare disease akin to macular degeneration, Erik Weihenmayer was sightless by age 13. Even so, he continued to pursue his dream of mountaineering, and he succeeded: In 2001 he became the first—and to date the only—blind climber to summit Mount Everest. Today he is climbing with the aid of a tool that allows him to “see” in a new way—with his tongue.
In normal vision, light hitting the retina provokes electrical impulses that the brain translates into images. What the tool, called the BrainPort, does is convert light into electrical impulses that stimulate the tongue instead of the retina. With more tactile nerve endings than any other part of the body except the lips, the tongue can discriminate two points spaced less than a millimeter apart. That degree of resolution is far beyond what the current BrainPort array, with only 611 electrodes, provides. But tests have shown the BrainPort delivers enough information for users like Erik to navigate with.
“Climbing!” Erik calls out to his partner. He places the device in his mouth, raises his head, and “surveys” the wall before him. Electrical impulses from the BrainPort become to him “a tactile image that I’m interpreting in space.” Erik reaches for a handheld slider that controls the zoom level and field of view of the BrainPort so that he can “see” one or two rock climbing holds on the wall some 2 or 3 feet above him. Then he slowly and deliberately raises his right hand, reaches up, grabs hold, and begins to pull himself up the wall. His feet secure, he hangs for a moment, adjusts the zoom again, then tentatively puts his hand out and above him again, this time missing the hold by half a foot. Judging the distance of an object in space is particularly difficult even with the device, Erik says. To help him figure out how far away something is, he sometimes waves his hand—an object of known size and distance—in front of himself first to get a sense of scale via the BrainPort array. The feedback helps him assess the distance of his next rock climbing hold.
Testing the BrainPort, Erik says, involves “learning to climb in a new, different way. I’m learning another language in the same way someone would be learning Braille or French for the first time. I’m figuring out how to map it spatially.” The challenges are significant. The BrainPort provides information in two dimensions, like a line drawing on a piece of paper, but the user’s brain must learn to translate this information into things like perspective, dimension, and location in space. When Erik first used the BrainPort, the images in his brain appeared as unidentifiable shapes and lines, but over time, through practice, his brain adapted, eventually translating the tactile sensations into recognizable patterns and symbols.
Erik has experimented with the tool at his home in Golden, Colorado. He could find and grab a coffee cup placed on the center island in his kitchen; play soccer with his daughter Emma in their large downstairs playroom, kicking the ball back and forth over some 15 feet; play rock-paper-scissors; and compete in a game of tic-tac-toe by identifying x’s and o’s on paper mounted on an easel. Later, the two moved out to the snowy front yard to play hide-and-seek. Eventually Erik could detect Emma’s movement as well as locate a snowman. Outdoor environments remain particularly challenging, though.
“Outside, shadows create lines, which appear on the tongue as objects in space,” Erik says. “For sighted people, the eyes and the brain quickly make adjustments, but the BrainPort doesn’t always keep pace with the shadows, and one has to learn to interpret these situations and contexts.”
Paul Bach-y-Rita, the cocreator of the system Weihenmayer is testing, challenged conventional wisdom when he proposed that “we see with the brain, not with the eyes.” He insisted that the brain was flexible enough to learn to interpret data coming in on a different kind of channel. Bach-y-Rita died in 2006, but Wicab—the company he cofounded in Middleton, Wisconsin, in 1998—continues to work on devices based on the idea that one sense can substitute for another. By 2004, Wicab had gained support for its research from the National Eye Institute, one of the federal agencies of the National Institutes of Health.
Wicab intends to create a BrainPort vision device for consumers, and over time the company has made it much less cumbersome. The prototype Erik first worked with in 2003 was bulky, comprising a webcam with a fixed field of view and a shoe box full of hardware that drove a basic 144-electrode array, all connected and run through a laptop computer. The current BrainPort prototype includes hardware contained in a belt-mounted 6-by-6-by-4-inch box, three sensitive cameras with a 90-degree field of view, and user-operated zoom and contrast control.
Still, despite significant improvements in size and portability, the current prototype is not cosmetically appealing or portable enough for ordinary use. Wicab plans eventually to integrate microcameras and microprocessors into eyeglass frames; a dental appliance will incorporate the tongue display, allowing the user to touch the display intentionally with the tongue, but also to pull it away and talk. Low-power radio frequency transceivers will link the cameras, processors, and tongue arrays; later on, wireless technology will connect these elements.
Other applications for the BrainPort include assisting military divers in challenging low-light and no-light search-and-rescue operations, says Rich Hogle, Wicab director of product development. “Imagine that you’re floating in a dense brown fog in daytime and visibility is such that…there’s no up, down, left, or right, and now your job is to swim 200 yards in a straight, level line, turn 180 degrees to the left, and swim back in a line parallel to the first. If you’re off by a few degrees in either direction, or too shallow or too deep, you might miss what you are looking for. Now imagine doing the same thing, in the same brown fog, at night.”
In research funded by the Defense Advanced Research Projects Agency, Wicab researchers used BrainPort technology to help military divers swim in a straight line in low-light and no-light conditions. “For this project we used a digital compass to provide directional information that was then displayed on the tongue as a tactile pattern,” Hogle says. “While swimming in the target direction, the diver’s eyes and ears were able to remain focused on the operational tasks at hand.”
Erik Weihenmayer hopes that the device will eventually benefit other members of the blind community, especially those who have never experienced vision. “I wonder how great one’s learning curve would be if you used this device two hours a day, every day. All of these blobs and circles and lines you would be able to learn and interpret. If you gave it to a blind child, his spatial understanding of the world would be through his tongue. It’s a compelling thought.”
The BrainPort provides a window into the vast complexity of our senses, Weihenmayer says. “I’m reminded how amazing the brain really is, with its incredible ability to differentiate all this information. It’s a trip to be able to ‘look’ at a hold and reach for it. For me, it’s beyond science. It’s a kind of vision.
