To qualify for full-fledged combat roles, autonomous robots would have to master the art of discrimination, recognizing schools, hospitals, and city centers and distinguishing a bomb-packing insurgent from a book-toting schoolboy before firing a shot. Inching closer to this goal is an upgraded version of the MQ-9 Reaper, a single-propeller aircraft typically armed with several guided missiles and bombs. In its current incarnation, the MQ-9 is remote controlled. A pod of cameras hangs beneath its “chin,” sending images to a satellite. From there the pictures are beamed to a remote pilot, who surveys the scene and uses a keyboard and joystick to issue commands controlling the MQ-9’s actions. A limitation here is scope. Current-generation aerial drones have a minuscule field of vision, showing just the area immediately below. Only by placing those views in a broader context—by looking at pictures of the overall terrain sent independently by satellite—can the pilot know what to do.

To overcome such limitations, a new imaging system for the Reaper, called Gorgon Stare, will bequeath the gift of wide-angle sight. Named for the mythical Gorgon sisters, whose stares turned anyone to stone, the system will capture minute detail and subtle motion over an area the size of a small town, according to Robert Marlin, Air Force deputy director of Intelligence, Surveillance, and Reconnaissance Capabilities. It will do so by stitching together data gathered by five high-resolution cameras and four night-vision cameras. Although many of the technical specs remain classified, Gorgon-equipped Reapers are scheduled to begin scanning wide swaths of terrain in active duty this fall. Up to 10 individual video feeds within each swath will enable commanders and analysts to track activity in individual buildings or along side streets. Some Air Force plans hint at future versions of Gorgon Stare tracking up to 96 individual scenes from a single Reaper drone.

Already the technology has inspired private companies to develop software for monitoring the vast rivers of data each Gorgon-enhanced Reaper will send back. The first-generation software will probably resemble the computer programs that department stores use to monitor shoplifting surveillance cameras, Air Force technical adviser Mike Welch says. The department store security guards on duty at any given time may be looking at a bank of 40 monitors displaying camera feeds from throughout the store. One or two people cannot possibly watch every monitor simultaneously, so stores use security software that automatically looks for movements in “high threat” regions of the store and then alerts the guards. “They have software that tips them off for things that don’t fit the model,” Welch explains. “If people are standing around the jewelry counter, for instance, it will notify the guard.”

Human analysts working with Gorgon Stare would monitor baseline video to discover the equivalent of the jewelry counter. The computer would make this monumental job tenable, notifying analysts when a certain level of human motion or a certain type of vehicle has been observed.

Innocent as this may sound, as the number of Reapers and the number of video feeds from each Reaper continue to rise, monitoring will need to be increasingly automated and complex, Singer says. Gorgon Stare pushes the Reaper and its successors farther down autonomy’s slippery slope. “You don’t have a fire hose shooting back data at you; you have the system decide what’s important to send back or not,” he says. “That’s autonomy.”

Rules of Robot Engagement
When a soldier decides whom to watch, target, or kill, he or she draws on a well-sorted set of rules of engagement. As we increasingly empower our machines to make those decisions, the problem of discrimination will be subsumed by a broader issue: robot ethics.

That is where Georgia Tech roboticist Ronald Arkin comes in. In 2009 he completed a three-year research project for the Army to create an autonomous military robot software system imbued with basic ethics. More sophisticated versions of the code might someday give a robot an artificial conscience, based on standard laws of war and on guidelines such as the Army Field Manual and the Geneva Conventions. This “ethical governor,” as Arkin calls it, would act like a judge reviewing every instance in which a military robot might use lethal force.

Arkin cites the wartime atrocities that human soldiers have committed in the heat of battle as a motivating factor behind his work. One of his friends was a lieutenant in the Vietnam War. “He mentioned that when they were in free-fire zones, the people in his unit fired at anything that moved,” Arkin recalls. “We can design robots that can do better than that, I am fully convinced.”

The current proof-of-concept version of Arkin’s ethical governor assumes that discrimination has been programmed into the machine’s software. The robot is permitted to fire only within human-designated kill zones, where it must minimize damage to civilian structures and other regions exempted by human operators. The program then weighs numerically what soldiers in battle weigh qualitatively. One part of the code, for instance, makes basic calculations about how much force should be used and under what circumstances. What Arkin calls the “proportionality algorithm” mathematically compares the military importance of a target (ranked on a scale of 0 to 5—a number that a human must provide) with the capability of every weapon the robot has and with every nearby place from which the robot can fire that weapon. In some situations the robot might have to reposition before it fires to ensure minimal collateral damage. In others the target might be ensconced within a school or crowd of civilians, creating a situation in which the algorithm prohibits firing under any circumstance at all.

Last year Arkin ran computerized battlefield simulations involving an armed, autonomous aerial drone, viewable on his Web site. Rules of the ethical governor were clear: For instance, restricted by blast radius, the drone could not launch a 500-pound laser-guided bomb on targets less than 2,000 feet from noncombatants—unless that target was critical, perhaps a top-level Al Qaeda operative. The governor also controlled the drone’s Hellfire missiles (allowed to hit targets 20 feet from noncombatants) and machine guns (1 foot).

Before the decade is out, some fighting force may field a military robot that can kill without a joystick operator behind a curtain elsewhere in the world.

In these simulations, the drone approaches kill zones with civilian structures such as an apartment building, a religious landmark, and a hospital. In one scenario, the simulated Reaper targets a gathering of enemy combatants at a funeral. But the ethical governor finds no weapon or firing position that ensures the safety of civilians and avoids the desecration of the cemetery, so the software orders the drone to hold its fire.

A second scenario has the Reaper flying into a kill zone with an enemy convoy driving between an apartment building and a religious landmark. This time the governor permits the drone to fire on the convoy, but only after first weighing the military value of the target against potential damage to civilian structures.

With further development, the ethical governor could enable military robots to use lethal force with a closer adherence to the laws of war than human soldiers achieve, Arkin argues. “And robotic systems do not have an inherent right to self-defense,” he says, so a robot could be used to approach an unknown person or vehicle without the haste or panic that a human soldier might feel. “These systems can solve problems, and we should not think of them as operating in the same way as human soldiers. They won’t.”

Arkin says his ethical governor is still in its early stages, so rudimentary that it cannot even be prototyped for testing in the field. But he also calls it naive to suppose that military robots will not gain in autonomy as technology improves. In his view, building smarter and better artificial ethics is crucial to keeping robot autonomy in check.

Tech planners like Arkin focus on near-term, plausible battlefield challenges. The military machines that they worry about are far removed from the famous sci-fi killer robots of the Terminator movies or Battle­star Galactica or the sophisticated, near-human machines of Isaac Asimov’s I, Robot and Philip K. Dick’s Do Androids Dream of Electric Sheep? Yet philosopher of technology Peter Asaro of the New School in New York City, cofounder of the International Committee for Robot Arms Control, thinks there are useful lessons to be learned from the fictional extremes. He worries that even if military robots with emerging autonomy are dumb by human standards, they may still be smart enough in their specialized domains to commit accidental massacres or even start accidental wars.

For instance, Asaro says, glitches in an autonomous aerial drone that lead to accidental missile strikes may not be easily distinguishable from a drone’s “intentional” missile strikes. The more politically tense the situation, the more likely that unpredictable military actions by autonomous drones could quickly descend into all-out warfare, Asaro believes.

Say, for instance, that Iran is testing its own autonomous aerial drone and fires on an American troop convoy just over the Afghanistan border. Even if Iran disavows the drone’s actions, no one may ever truly establish intent. Iran could insist it was a glitch, while hawks in the United States would have all the casus belli they need to launch a new Middle East war.

A battle of robot against robot might also erupt. With up to 50 nations around the world developing military robots, says physicist Jürgen Altmann of Dortmund Technical University in Germany, opposing aerial drones could ultimately square off against each other. Operating drones by remote control via satellite adds at least a half-second delay, he notes, so the pressure to switch to quick-draw autonomous mode would be strong, if only to ensure having the upper hand.

Altmann describes another hypothetical situation in which Chinese and American drones encounter each other in the North Pacific off the coast of Guam. Such a confrontation might not remain restricted to unmanned drones for long. “You might have a solar reflection or something that is mistaken for a first shot,” he says. “If you have automatic reactions, you might stumble into a shooting war through any kind of unclear event.”

Robot Arms Control
Given the risks, many see robot autonomy as a genie in a bottle, best kept contained. Last September Altmann and ethicist Robert Sparrow from Monash University in Melbourne, Australia, traveled to the home of Sheffield University robotics researcher Noel Sharkey in the U.K. (Asaro attended electronically.) Sharkey had convened this small conference to allow 48 hours of intense debate and deliberation about the problem of autonomous military robots.

Bleary-eyed from one long day and two very late nights—and hastened by the need to catch their respective trains in the early afternoon—the four participants agreed to draft a founding document for what became the International Committee for Robot Arms Control. Inspired in part by the Nobel Peace Prize–winning anti–land mine movement, all present agreed that militaries around the world are asking too little and moving too quickly. “Machines should not be allowed to make the decision to kill people,” the committee summed up in its one-page position paper.

Sharkey frequently speaks to military officials around the world about autonomous lethal robots. At the Baltic Defense College in Tartu, Estonia, this past February, he took a straw poll of the military officers who had come to hear him speak. Fifty-eight of the 60, he said, raised their hands when asked if they would like to halt further development of armed autonomous robots. “The military people I talk to are as concerned as anybody,” he says.

At the same time, there is no question that unmanned machines like the PackBot and the Predator drone have been extremely useful in America’s most recent conflicts. Robot autonomy could be even more valuable, for both military and civilian applications: for search and rescue, disarming bombs, or delivering better and faster medical care. Most American fatalities in Iraq and Afghanistan, for instance, come from improvised explosive devices (IEDs). “Nobody’s making an autonomous bomb-finding robot,” Sharkey says. “At the moment, the machinery for doing that is too large to fit on a robot. But that’s where the money should go.”

Asaro proposes that the next step be an international treaty, like the Ottawa Mine Ban Treaty, to regulate the technology. “We should be worried about who is going to be harmed. We need international standards and a test to prove the technology can truly discriminate,” he says. Arkin agrees, noting that a functioning ethical governor to regulate robot behavior on the battlefield is at least a decade away.

But it is not even clear that ethical behavior is a programmable skill. The spotty record of artificial intelligence research does not inspire confidence. “If I were going to speak to the robotics and artificial intelligence people,” Colonel Sullivan says, “I would ask, ‘How will they build software to scratch that gut instinct or sixth sense?’ Combat is not black-and-white.”