Tech companies are eyeing the next frontier: the human face. Should you desire, you can now superimpose any variety of animal snouts onto a video of yourself in real time. If you choose to hemorrhage money on the new iPhone X, you can unlock your smartphone with a glance. At a KFC location in Hangzhou, China, you can even pay for a chicken sandwich by smiling at a camera. And at least one in four police departments in the US have access to facial recognition software to help them identify suspects.
But the tech isn’t perfect. Your iPhone X might not always unlock; a cop might arrest the wrong person. In order for software to always recognize your face as you, an entire sequence of algorithms has to work. First, the software has to be able to determine whether an image has a face in it at all. If you’re a cop trying to find a missing kid in a photo of a crowd, you might want the software to sort the faces by age. And ultimately, you need an algorithm that can compare each face with another photo in a database, perhaps with different lighting and at a different angle, and determine whether they’re the same person.
To improve these algorithms, researchers have found themselves using the tools of pollsters and social scientists: demographics. When they teach face recognition software about race, gender, and age, it can often perform certain tasks better. “This is not a surprising result,” says biometrics researcher Anil Jain of Michigan State University, “that if you model subpopulations separately you’ll get better results.” With better algorithms, maybe that cop won’t arrest the wrong person. Great news for everybody, right?
It’s not so simple. Demographic data may contribute to algorithms’ accuracy, but it also complicates their use.
Take a recent example. Researchers based at the University of Surrey in the UK and Jiangnan University in China were trying to improve an algorithm used in specific facial recognition applications. The algorithm, based on something called a 3-D morphable model, digitally converts a selfie into a 3-D head in less than a second. Model in hand, you can use it rotate the angle of someone’s selfie, for example, to compare it to another photograph. The iPhone X and Snapchat use similar 3-D models.
The researchers gave their algorithm some basic instructions: Here’s a template of a head, and here’s the ability to stretch or compress it to get the 2-D image to drape over it as smoothly as possible. The template they used is essentially the average human face—average nose length, average pupil distance, average cheek diameter, calculated from 3-D scans they took of real people. When people made these models in the past, it was hard to collect a lot of scans because they’re time-consuming. So frequently, they’d just lump all their data together and calculate an average face, regardless of race, gender, or age.
The group used a database of 942 faces—3-D scans collected in the UK and in China—to make their template. But instead of calculating the average of all 942 faces at once, they categorized the face data by race. They made separate templates for each race—an average Asian face, white face, and black face, and based their algorithm on these three templates. And even though they had only 10 scans of black faces—they had 100 white faces and over 800 Asian faces—they found that their algorithm generated a 3-D model that matched a real person’s head better than the previous one-template model.
“It’s not only for race,” says computer scientist Zhenhua Feng of the University of Surrey. “If you have a model for an infant, you can construct an infant’s 3-D face better. If you have a model for an old person, you can construct that type of 3-D face better.” So if you teach biometric software explicitly about social categories, it does a better job.
Feng’s particular 3-D models are a niche algorithm in facial recognition, says Jain—the trendy algorithms right now use 2-D photos because 3-D face data is hard to work with. But other more widespread techniques also lump people into categories to improve their performance. A more common 3-D face model, known as a person-specific model, also often uses face templates. Depending on whether the person in the picture is a man, woman, infant, or an elderly person, the algorithm will start with a different template. For specific 2-D machine learning algorithms that verify that two photographs contain the same person, researchers have demonstrated that if you break down different appearance attributes—gender, race, but also eye color, expression—it will also perform more accurately.
So if you teach an algorithm about race, does that make it racist? Not necessarily, says sociologist Alondra Nelson of Columbia University, who studies the ethics of new technologies. Social scientists categorize data using demographic information all the time, in response to how society has already structured itself. For example, sociologists often analyze behaviors along gender or racial lines. “We live in a world that uses race for everything,” says Nelson. “I don’t understand the argument that we’re not supposed to here.” Existing databases—the FBI’s face depository, and the census—already stick people in predetermined boxes, so if you want an algorithm to work with these databases, you’ll have to use those categories.
However, Nelson points out, it’s important that computer scientists think through why they’ve chosen to use race over other categories. It’s possible that other variables with less potential for discrimination or bias would be just as effective.“Would it be OK to pick categories like, blue eyes, brown eyes, thin nose, not thin nose, or whatever—and not have it to do with race at all?” says Nelson.
Researchers need to imagine the possible applications of their work, particularly the ones that governments or institutions of power might use, says Nelson. Last year, the FBI released surveillance footage they took to monitor Black Lives Matter protests in Baltimore—whose state police department has been using facial recognition software since 2011. “As this work gets more technically complicated, it falls on researchers not just to do the technical work, but the ethical work as well,” Nelson says. In other words, the software in Snapchat—how could the cops use it?
