It was Just after midnight in New York when police chased down and arrested
the suspect wanted in a potential hate crime. A gay man had been shot and killed with a silver revolver. The suspect was cooperative to a point. He gave them the silver revolver in his holster. He also gave them an ID.
Then he clammed up. When they brought him to the precinct to book him, the ID turned out to be fake. He wouldn’t tell them his real name. They couldn’t take his fingerprints.
By this time, it was just after 7 a.m. The officers called Edwin Coello, the sergeant who has led the New York Police Department’s Facial Identification Section since it was formed in late 2011. It was a Saturday, and Coello was still in his robe at home, but he pulled up a scan of the ID on his laptop and started working.
A guy cracking a case in his bathrobe sounds like something out of a cop show. On TV, police officers use technology as a kind of magic detective’s aid, pumping out important clues on demand. In the office, on a normal day, watching actual police detectives use actual technology on actual cases presents a more complicated picture. “It’s far from The Bourne Supremacy,” Coello says.
In the unit’s office on the ninth floor of One Police Plaza, a Brutalist monolith in south Manhattan, banks of 55-inch LED monitors hang from the wall, an ever-changing art gallery of cases. But most of the work happens in the detectives’ cubicles, clustered in the middle of the room.
Each has three monitors. A detective uses a mouse to rotate a face on her screen to the same full-frontal angle as a mug shot. The photo is distorted, taken from a security camera with a fish-eye lens. She’ll try it this way against the arrest database, but if she gets no matches, she will begin to adjust the fish-eye ratios so the person’s face will have normal proportions, hoping for clues to an identity. Another detective captures a frame from a cell phone video from a crime scene. He’ll also run that through the mug shot database. One screen shows a wire-frame rendering of a face in 3-D; that detective is working to build out a full facial image based on a profile view.
What's in a Face?
With most facial recognition technology, faces don’t look like faces at all. Instead, streams of numbers represent hairlines, eyebrows, nose structure, jaw lines, ears, scars or moles, as well as the head’s pitch, yaw and roll (head angle, whether the chin is up or down, and how far off-center the face is). The NYPD’s software accounts for more than two dozen factors in all. Algorithms, or recipes for how computers solve a problem, use the numbers to try to determine whose face it is. A simple change of expression can throw off the numbers in ways that create real problems for an algorithm.
It can take hours or even days to prepare a photo for comparison against a database. But on that Saturday morning working from home, Coello had a straight-on photo with a neutral expression in poor lighting. “I had to enhance it because the image was a little blotchy,” he says.
After a couple of minutes, Coello asked the investigator, “Can you just take a look at him and tell me: Does he have a scar or mark on his face?” There were no marks on the face in the ID, but there was a scar on one of several hundred faces the database offered as a match. The investigator said the suspect has a scar. Coello told the detectives that a possible match is Elliot Morales, who had been arrested six times, once on suspicion of attempted murder. The work took about an hour, and Coello was back to hanging out with his kids on a Saturday.
More and more, we’re seeing facial recognition technology emerge as part of crime-fighting, as police departments hear about successes like those in New York. Outside of New York, such as Pinellas County, Fla., where the software has been used for 14 years, it’s helped capture criminals with outstanding warrants during routine traffic stops.
The NYPD has used facial recognition technology in more than 8,000 cases, and it found 2,000 possible matches in its arrest database. Of those, more than 1,000 led to arrests. One notable case in 2012 involved a shooting at a barbershop, where a man fired a shotgun at an acquaintance getting a haircut. The victim and other witnesses knew the man only as “Armani,” but one of them had Armani’s picture on his Facebook page. Police used that photo to look for a match in their database, found one, and made the arrest.
Facial recognition software is becoming common in motor vehicle registries, too, as more than three dozen states use the technology to look for fake driver’s licenses and identify people during investigations.
At the federal level, the Diplomatic Security Service, a unit of the State Department, used a newly installed facial recognition system in 2014 to find Neil Stammer, who skipped bail in 2000 on charges including sex with a male minor. The agency was testing the system by running it against the latest FBI wanted posters; by happenstance, Stammer’s mug appeared on one of them. He’d been living under an assumed name in Nepal, where he left a trail by regularly renewing his tourist visa.
In September 2014, the FBI unveiled Next Generation Identification, a database expected to have 52 million records in it, created by merging multiple criminal and civil photo and fingerprint databases from the U.S. and abroad, all in the name of fighting crime and terrorism.
Big Brother, and Siblings
All this facial recognition technology in the hands of government raises unavoidable questions about Big Brother-like powers. Only it could be even more pervasive than Orwell imagined. There are surveillance cameras all over the place in major cities. “The risk is that the government can track people as they move from place to place,” says Jennifer Lynch, senior staff attorney at the Electronic Frontier Foundation. It hasn’t happened yet as far as civil liberties advocates can determine from public legal records, but there are no specific rules or regulations against it, she says.
And Big Brother might have siblings. Retailers are already using facial recognition systems to target shoplifters if they hit a store, or chain of stores, more than once. That’s similar to how casinos use the technology to identify known card counters, whom security guards can then remove. Some bars used a mobile phone app (before the developer went out of business) that scanned people’s faces as they entered. The app determined if bar-goers were male or female, their relative age, and reported the ratio to app users deciding where to head out for the night.
The NYPD cites, repeatedly and without irony, concern for civil liberties as a reason why it only uses the mug shots in its arrest database to look for suspects. But some law enforcement agencies also link to driver’s license databases in their states, and the FBI’s new database is open to state and local law enforcement.
There has not yet been a prominent case of misidentification, or false positive, using a facial recognition system. But give it time; issues still occur with fingerprinting, a much older and generally more accurate form of identification than facial recognition, which is sometimes called “faceprinting.” After the 2004 Madrid train bombings, the FBI linked Brandon Mayfield, an American attorney, to the bombings through a fingerprint match that proved inaccurate. He spent two weeks in prison and later won a $2 million court settlement and a formal apology from the FBI.
Coello stresses that New York police do not use facial recognition matches as conclusive evidence to arrest someone. “It’s only a lead for detectives,” he says. “We point them in the right direction.” The people who work in facial recognition are all detectives, and they do legwork beyond the photos, conducting detailed searches of a possible suspect’s background, like their address, to aid the investigation. “No one is going to go four towns over to hold up a liquor store,” Coello says.
Even with clear matches, though, the facial recognition team says the person is only a possible suspect. The department says it has misidentified someone via the technology just five times, most recently in March 2012.
“It’s just a tool. It’s not DNA, and it’s not fingerprints,” says Stephen Capasso, the former commanding officer of New York City’s Real Time Crime Center, which includes the facial recognition unit. Still, “I think our usage of facial recognition is going to be increasing.”
Most of us encounter facial recognition in perfectly law-abiding modes, like photo tags on Facebook and on photo apps like Google Photos, where software algorithms parse our pictures and suggest names for the people in them. Facebook launched its photo-tagging tool in late 2010, and it’s become a routine feature for many users. This is certainly the first mass consumer use of facial recognition. It likely won’t be the last.
How We Got Here
Travelers, for instance, might encounter facial recognition algorithms at airports. In Australia at the end of 2013, P. Jonathon Phillips walked through SmartGate, an automated border control system being used in Australia’s eight major international airports to speed customs processing for people from eight countries, including the U.S.
Phillips put his passport in the kiosk and looked at a camera, which automatically matched his face with the image on the passport. He was through SmartGate in five minutes. He knew about the system, but still, “I was amazed when I saw this!” he says. “I’ve been in the facial recognition field for 23 years. We started out with ‘can you recognize?’ algorithms. When you go someplace and it actually happens …”
Phillips is arguably the most influential scientist in facial recognition. He started his work in 1993, launching the FERET (Face Recognition Technology) program for the Army Research Laboratory, the first such program. Back then, they were testing algorithms against a database of about 1,200 faces, mostly college student volunteers from George Mason University. He’s now an electronic engineer at the National Institute of Standards and Technology, and he manages NIST’s facial recognition challenges.
When he started, verifying passport photos presented a difficult problem. Now, many facial recognition algorithms are better than humans when it comes to recognizing a person looking straight ahead under good lighting conditions.
Facial recognition algorithms don’t “see” anything, of course. Faces and their features are broken down into strings of numbers representing individual pixels, their colors and their place on what will mathematically correspond to a face. Algorithms must first find a face, and then find the eyes and other features that human brains take in at once. One early technique came via a linear algebra representation called eigenvectors, which let researchers compare similar objects as long as they are precisely aligned. Think driver’s license and passport photos, or mug shots, which feature a face looking straight ahead. Researchers used these techniques to create eigenfaces, which to human eyes look ghostlike, but they give algorithms a reference representation of a face to compare with a new face.
It helps the technology that faces are relatively straightforward to analyze. Eyes and mouths are in the same places consistently, and face shapes don’t vary much — you’ll never find someone with a face shaped like a square, star or a hexagon. By the mid-1990s, facial recognition was a hot technology, and several startups formed to commercialize it.
“That was just fun, the first stage of a new technology sprouting up,” says Brian Martin, senior director of research and technology at MorphoTrust, the dominant provider of facial recognition software to government and law enforcement. In 1998, he received a Ph.D. in condensed matter physics from the University of Pittsburgh. A year earlier, he started working at Visionics, an early facial recognition startup. Its first product? A biometric screen saver that used your face as your computer password. Martin says it had two big selling points: You didn’t have to remember your password, and it would take pictures of anyone who tried to break into your computer. But that day’s low-resolution cameras meant the technology was especially prone to issues with image quality.
Martin says accuracy improved when researchers started to use what’s called local feature, in which algorithms don’t just use the whole face, but also patches of it, like the shape of eyebrows and the width of the nose. That makes the software less prone to stumbling over expression changes. Around 2005, researchers began applying machine learning techniques to their algorithms, to train them to learn how to match sets of features more and more accurately.
Today the big push is in “deep learning” — building artificial intelligence algorithms inspired by the brain’s neural connections. Our neurons connect to each other through trillions of synapses, which are like nodes on a computer network. These form individual connections influenced by the number of times each neuron communicates with another neuron. Neural networks also consist of connections adjusted by exposure, in effect learning from developing patterns. In facial recognition, these patterns emerge after being “shown” thousands upon thousands of photos. The algorithms mathematically recognize when a face shares enough characteristics to likely be the same as a face in different lighting or with a different expression. Neural network theory isn’t much different than it was in the 1990s, but this century’s explosion in cheap computing power and availability of data lets researchers take full advantage of the theories.
Once they’ve trained their algorithms, researchers often test them against publicly available benchmarks. The most popular is a database of images gathered by Erik Learned-Miller, a computer scientist at the University of Massachusetts Amherst, and his colleagues. In 2003, Learned-Miller was a postdoctoral researcher at the University of California, Berkeley, working with a database called Faces in the Wild, developed by David Forsyth, Tamara Berg and others. They pulled tens of thousands of different faces from Web-based news sites (“the wild”) to train a facial recognition algorithm they were working on.
The goal was to train the algorithm to automatically label the people in the photos. After a paper on the algorithm was published in 2004, Learned-Miller began getting requests from facial recognition researchers who wanted access to the database for their own work. In 2007, he and a colleague cleaned up the database, getting rid of duplicates and miscaptioned photos, and released it as “Labeled Faces in the Wild,” with 13,233 different photos of 5,749 people. The timing was excellent — facial recognition algorithms were getting good at identifying faces in controlled environments, such as passport photos. Researchers wanted something more challenging, and this database fit the bill.
It has since been cited in more than 1,100 papers. In the past eight years, more than 60 research groups have sent the results of their algorithms against the benchmark for Learned-Miller to post on the Labeled Faces site. Facebook’s DeepFace algorithm had the best performance for a brief period last year, only to be passed by a Chinese company, Megvii, whose Face++ algorithm hit 99.5 percent accuracy.
Facing the Future
Those kinds of numbers suggest Labeled Faces in the Wild has been tamed. Learned-Miller says it’s time for facial recognition researchers to move on to new problems. There are still plenty of those. Computers have a hard time recognizing faces in less-than-ideal lighting conditions, or when analyzing faces at more than a 40-degree angle from straight on.
Failure rates can run as high as 2 in 10 for everyday “point and shoot” photos, according to Phillips, compared with 1 in 1,000 for mug shots. NIST’s current facial recognition project, dubbed Point-and-Shoot Face Recognition Challenge (PaSC), involves a dataset of 9,376 still images and 2,802 videos. Labeled Faces in the Wild uses photos of well-known people taken by professional photographers. The PaSC, launched in October 2013, focuses on the rest of the world’s photos: those taken on cell phones or simple digital cameras. These can be poorly staged and blurry.
At its core, facial recognition is an artificial intelligence challenge, albeit one that is becoming less challenging. Phillips says knowing how humans recognize faces in these conditions will help develop the algorithms. Most algorithms focus on the center of the face where humans use lots of different cues about a person, like the part of their hair. We are better — after spending a morning with someone, most of us will recognize that person’s face consistently, in most lighting conditions, at most angles. The algorithms need intensive training, the deep learning that takes advantage of computational speed and pattern-matching.
Some researchers, like Learned-Miller, have tried very different approaches. When I visited his lab at UMass Amherst, his current tool was, oddly, an old-fashioned world globe. The globe was not quirk, but work — it was a stand-in for a human head. He was working on an algorithm that could recognize the rotation and position of the globe. This could help make algorithms better at recognizing the pose of a head, for example: Is it straight on, or in profile? His approach would be called unsupervised learning. (Feeding the algorithm many pictures of faces in different poses would be called supervised learning.)
On the day of my visit, Learned-Miller and then-grad student Cheni Chadowitz looked through data generated by the algorithm, written in the computer program MATLAB. Chadowitz had adapted an earlier algorithm written by Learned-Miller that only recognized faces in images that showed them straight on. Learned-Miller stared at a scatter plot, which showed the improved algorithm was having some success at taking different images of a geographic feature, like the Horn of Africa, and putting it at the correct latitude and longitude.
The globe project was temporarily put on hold after Chadowitz graduated, but Learned-Miller says he’s waiting for the right student to continue working on it. For now, Learned-Miller is focusing on solving the problem of making facial representations that don’t take up a huge amount of memory. He’s working on an algorithm that combines many images into a “mental model” that captures all the information about a face that a video might, but is much smaller.
While Learned-Miller’s basic research may not impact the market for 10 years or more, other researchers work on a shorter timeline. “A lot of people in computer vision are really excited about building things that work, and it’s difficult to build things that work well,” he says. “Often we’ll build something that works 90 percent of the time, but if you put it out in the world and it fails one out of 10 times, in the consumer world, that’s unacceptable.”
Putting on Their Work Face
While facial recognition companies have emerged in the past, only to fail or struggle, the time seems ripe for commercialization. It’s a nearly $3 billion business, led by companies like MorphoTrust, NEC and Cognitec, which see their biggest sales from government and corporate security. But the market is expected to double by 2020.
Some of that growth will come from expansion in traditional markets as more law enforcement agencies adopt the technology. New kinds of consumer applications are also emerging. Facial recognition software can be used for other things besides identifying people. It’s also used to search for images of people with certain features or to track head movement and facial expression. Megvii’s Face++ is being used in China by dating services and even for playing video games. Companies are using a facial recognition programming interface from Kairos, a Miami-based provider of a facial recognition interface, for things like time management software, health care management and in amusement parks that want to sell photos to visitors.
There is also a move to blend facial recognition with a technology called facial analysis, which uses a person’s facial expression to predict mood and even diagnose certain illnesses.
Such applications raise substantial privacy concerns: Imagine insurers setting rates based on what your photo suggests about your health. The Electronic Frontier Foundation’s Lynch, pointing to people suspected of shoplifting, says no rules exist to prevent companies from sharing information. “If you shoplift and you’re caught by security guards, the store has the right to exclude you,” she notes. “But there is the potential for this to trail you from store to store.” That already happens in the casino business.
Our privacy laws are also built around our own efforts at self-protection. The Do Not Call Registry, for instance, kicks in only if you register your number in an official database. Facial recognition is different — cameras, often installed by police departments, constantly record us just for walking down a public street.
“Real taxpayer money is being spent to allow law enforcement to point a camera at protesters and be able to identify them by name,” says Alvaro Bedoya, executive director of Georgetown Law’s Center on Privacy and Technology. “We need to think about whether that’s a world we want.”
The flip side of this comes from Sgt. Coello of the NYPD. He says facial recognition is terrifically helpful, in practical ways. Detectives used to have to go door to door with an image trying to find out if someone knew the person in the photo. “We don’t need to do that now,” he says. “You get us the photo, we’ll do the rest [in the database].”
Bedoya acknowledges there are specific ways in which the technology could benefit the public. But he worries that it might inhibit public demonstrations, or let strangers and companies snap a photo of us and learn our names, occupations and addresses. An app called NameTag can already do some of this. Its maker, Las Vegas software developer FacialNetwork.com, last year launched an app called CreepShield tied to databases containing photos of half a million registered sex offenders. In June, Bedoya and eight other consumer advocates walked out of discussions on privacy guidelines led by the National Telecommunications and Information Administration, part of the Department of Commerce, citing lack of incentive for businesses to give people the right to consent to having their faces recognized.
There have been no court cases involving the civil liberties impact of facial recognition, partly because we don’t yet have evidence that the government or companies are misusing the technology. Texas and Illinois are the only states that have passed laws regulating commercial use of facial recognition. Facebook was sued in April for violating privacy restrictions in Illinois’ law; if that case makes it to trial, it will set a precedent.
Technologies are famously neutral; it’s people who decide whether to use them for good or ill. With facial recognition poised to become a far more widespread tool, we have a choice to make about how and when to blindfold it.
[This article originally appeared in print as, "Face Time."]
