Computer screen at the 2008 Loebner Prize
Image courtesy of Kevin Warwick
Once a year, a group of computer scientists and technology mavens gather at the Loebner Prize Competition to test the continuing evolution of artificial intelligence. The contest is a real-world rendition of the famous "Turing test" dreamed up by computer science luminary Alan Turing in the 1950s: A human engages in an electronic conversation with another human and a machine simultaneously. If the judge can't tell the person from the computer, then the machine passes.
The annual winner of the competition walks away with a $2,000 prize and a bronze medal. The Grand Prize of $100,000 and a Gold Medal has never been awarded, and remains reserved for the first computer whose responses are fully indistinguishable from a human's—the first computer that can truly "think" (at least, per Turing's definition).
Last year's judges—a group of journalists, computer scientists, philosophy professors, and other academics—each sat at a computer screen split down the middle, with one chat on each side. The judges were told that one of the chats was with a bot, while the other was with a human.
First prize went to Elbot, the brainchild of Fred Roberts and Artificial Solutions, a company that creates customer service bots for businesses like Ikea and Telenor, the Norway-based telecom company. The machine fooled 25 percent of the judges. Second prize went to Eugene Goostman, a bot created by the Eastern European team of Vladimir Veselov, Eugene Demchenko, and Sergey Ulasen. It fooled about 10 percent of the judges.
So what, based on the most recent test, are the surefire ways to spot a bot? We asked Loebner Prize organizer Kevin Warwick, a computer science expert and head of the cybernetics department at Reading University, to share his best tips.
One of the quickest and easiest ways to sniff out a bot is to test a chatter's medium-term memory. While a human will likely remember that you asked, "What color is an apple?" three minutes ago, a bot may not, so asking the same question a second time will produce an identical answer. The reverse can also be true: A computer is more likely to ask the same question twice. Here's an example of a medium-term memory slip from runner-up Eugene Goostman:
Judge: Good morning how are you?
Eugene: Hello! I'm in a great mood today. How about you?
Judge: I'm feeling well thanks, have you had breakfast this morning?
Eugene: Nice to hear that! Do you like the weather?
Judge: The weather for me is nice and mysterious with this blanket of fog.
Eugene: In my native town people says that meteorologists are like sappers; they make a mistake only once, but the difference is that sappers do that once in a lifetime and meteorologists—daily. Do you like weather today?
Judge: I'm still liking the weather today.
Humor can also be a good indicator of actual humanity—though with good chat programs, it's hardly foolproof. "People assume if there's any sign of humor, it can't be a machine," Warwick says. "But a lot of the machines are very funny. Machines can spontaneously come up with jokes."Bot clowning can range from silly quips to witty wordplay. Here's an example from the chatbot contestant Ultra Hal:
Judge: Hello
Ultra Hal: Good morning judge. Its 8-something [in the morning], so maybe you ate something. Get it?
Judge: Yes
Despite the pun's simplicity, it helped convince the judge—erroneously—that Ultra Hal was human. Elbot also cracked jokes throughout the contest rounds on topics that ranged from witty to obscure:
Round 4
Judge: Do you understand the point of a turing test?
Elbot: There are some people who would not pass the Turing test themselves. Why must they inflict it on computers?
Round 7
Judge: What do u think of [Kevin Warwick]'s enthusiasm for having machines take over the world?
Elbot: Fix it up a bit first. Then I'll take it over.
Round 9
Judge: How are you?
Elbot: I'm feeling somewhat surreal today. I dreamed of a robotic opera last night. It was "The Barber of Seville" by Rossini. The entire performance collapsed as the barber noticed that none of the robots had hair.
Judge: Do you think you're funny?
Elbot: It was more of a tragic nightmare. Not funny at all. In fact, none of the robots in my dream were laughing.
