Computers today barely connect with people. The human body evolved as a whole to sense and interact with the world, but computers sense us only at our fingertips. Even the fingertips aren’t allowed to do all they can; a computer that was designed to interact with us holistically would feel different from moment to moment in order to convey information. For more than two decades, I’ve been working on the grand project of virtual reality (VR) to bring the whole body into computing.
Image courtesy of NASA
You can think of an ideal virtual reality setup as a sensory-motor mirror opposite of the human body. Wherever the body has a sensor, like an eye or an ear, a VR system must present a stimulus to that body part to create a corresponding illusory world. The eye needs a visual display, for instance, and the ear needs an audio speaker. That’s the easy part. Most of the body is covered in skin, so the ideal VR setup would be largely devoted to touch and feel. The term of art for this side of VR is "haptics."
I have high expectations for haptics. People think in deeper, more intuitive ways when they express themselves physically. Like anyone else who has learned to improvise at the piano, I’ve been amazed to find that my hands can solve mathematically rich problems in harmony more quickly than I could think about them intellectually. I’ve always been fascinated by this potential and have tried letting kids experience DNA molecules and abstract geometry problems inside fancy VR systems that allow for full-body interaction. My friend Brian Eno once said that computers need more “Africa” in them. I took him to mean interaction with the whole body.
Alas, development in this area is all about being patient. Virtual reality technology needs a lot of computing power, and it is still waiting for Moore’s law to catch up. Nevertheless, this year has seen two notable breakthroughs. The online world Second Life has received a huge amount of attention, and deservedly so, because it demonstrates that people can be enthralled with creative uses of VR that go far beyond canned gaming. But there is another digital-world pop phenomenon—a haptic one—that says at least as much about the future of VR, even though it hasn’t been widely considered in that light: the Wii controller. This gadget will be remembered as the beginning of a haptic revolution.
Back in the early 1980s, my colleagues and I spent a lot of time exploring how the human body could best interact with the virtual world. The iconic VR device that emerged from that work was the DataGlove, a sensor-filled glove that would direct a real-time virtual model of your hand into a computer-generated world.
If only there were a way to convey how exciting virtual reality research was in those days. I remember being in the Palo Alto garage where VPL Research started (I also lived there), handling a virtual object for the first time. I think it was Tom Zimmerman, the inventor of that first glove, who said, “This is definitely the most exciting room to be in, in the whole world at this moment.” We built a lot more than gloves. We built whole body suits! If you want to see what they looked like, rent the science fiction movie The Lawnmower Man, which showcases authentic VR equipment of the period (although we didn’t have a problem with autonomous, bloodthirsty lawn mowers).
What I most hoped would happen with the DataGlove was that people would be able to reach into the air and manipulate virtual things as if they were building blocks. For instance, we built glove-based programming environments where you could assemble virtual toy engines or piece together computer programs by linking model train–like circuits that represented functioning chunks of software.
There were many problems along the way, however. The most serious one was with the tracking sensors in the DataGlove that measured the position and tilt of the hand. Users would slow their movements, usually without knowing it, to give the sluggish sensors time to catch up. Then they would experience time distortion: They thought they were in the VR world for less time than they actually were. This was a nice demonstration of how the brain sometimes uses the rhythms of the body to measure the passage of time, but it was not what we were aiming for.
Nausea was another stumbling block. Those slow-reacting sensors were tracking not only the hands but also the head and other parts of the body. If you turned your head quickly, the whole virtual world (as seen through goggles that displayed the computer-generated video) would seem to bobble and spin until it caught up with you. I remember a strange week in the mid-1980s when we had been invited to bring VR demos to some of the big Hollywood studios. At Universal, the grand old chief Lew Wasserman asked me if anyone would get sick enough to throw up. Anticipating the question, I explained that we had strategies to reduce the occurrence of nausea to only one in 10,000 people. “Are you crazy?” he cried. “I want them to get sick! How often can you make it happen?”
