Art historians and critics have spent countless hours analyzing the emotions of famous portrait subjects, from the mystery of Mona Lisa’s smile and the anguish of Picasso’s “Weeping Woman” to the tortured angst in Munch’s “The Scream.” This process of reading feelings in faces has long been considered an intuitive rather than scientific one — but perhaps artificial intelligence could soon help us settle these questions once and for all.
Microsoft’s Project Oxford photo research division has just released a new demo of a tool that detects emotions in photographed faces using machine-learning techniques. When you upload a picture to the site, the software scans the subject’s face and attempts to read the feelings behind their expression. It then presents a scorecard for each subject, ranking them on eight emotions: anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise.
The site does offer a disclaimer: “Recognition is experimental, and not always accurate.” Though they were developed after years of research, these robots are still learning, and their emotional vocabulary is pretty limited. Still, could this app offer any new insights into the emotional lives of famous portrait subjects? To test it out, we ran five artworks (admittedly not all photographs) through the app. It was a weird exercise in assigning numerical values to things usually considered unquantifiable. Here are some feelings in art from a robot’s perspective; see how its analysis compares to your own real, live human impressions.
Leonardo da Vinci, Mona Lisa (c. 1503)
This isn’t the first time the Mona Lisa has been subject to the scrutiny of emotion-recognition technology: back in 2005, computers at the University of Amsterdam confirmed longstanding human suspicions that this most famous of sitters is “mainly happy.” Microsoft’s new app agrees — the only categories in which the Mona Lisa received ratings higher than zero were “happiness” (0.43) and “neutral” (0.55). Before the advent of such advanced technology, even emotionally illiterate viewers could come to a similar conclusion by reading the painting’s Italian title, “La Gioconda,” which, in addition to being a play on the presumed sitter’s name (Lisa del Giocondo), literally translates to “jocund,” or “happy.”
Richard Avedon, “Karl Rove,” from Portraits of Power (2004)
Former White House Deputy Chief of Staff Karl Rove, in a photo from Richard Avedon’s series Portraits of Power, was rated the happiest of any subject we ran through the app: his smug, self-satisfied smirk earned him a nearly 100% happiness score. Microsoft’s technology failed to detect any of the politician’s warmongering bile. But Rove himself wasn’t very happy when he saw this portrait; he said it “makes me look like a complete idiot” and accused Avedon of being “an elitist snob who deliberately set me up.”
Dorothea Lange, “Migrant Mother” (1936)
Microsoft’s robot did manage to pick up on the sadness in the face of Florence Thompson, a “destitute pea-picker” and 32-year-old mother of seven in Depression-era California, shot by Dorothea Lange in 1936. But the unfeeling AI suggests that this vast melancholy can be quantified, giving it a neat little score of 0.13.
Cindy Sherman, “Untitled #414” (2003)
What emotion could Cindy Sherman’s super creepy, painted-on grin possibly signify? Happiness, says Microsoft’s still-learning robot. The only other subject it rated happier than this deranged evil clown was Karl Rove. Go figure.
Johannes Vermeer, “Girl with a Pearl Earring” (c. 1665)
Vermeer’s “Girl with a Pearl Earring” is often called “The Mona Lisa of the North” and described as enigmatic and mysterious. “The more you look at her the more the expression changes,” Melissa Buron, a curator at the De Young Museum, once said of the painting’s subject. “At first she seems caught in a moment, startled. And then she seems to be responding to a question.” Microsoft’s app doesn’t quite get that specific, assigning “Girl” 85% neutrality and 13% happiness, with tiny traces of sadness, surprise, fear, contempt, and anger lurking beneath.
There are plenty of fields in which emotion-detecting software like this might have practical uses. Marketers could use it to gauge people’s reactions to products, Microsoft’s Allison Linn writes, or to create consumer tools, such as a messaging app, that offers up “different options based on what emotion it recognizes in a photo.” And creepily, in our surveillance state, security professionals could use this tech to help keep an eye on angry or contemptuous-looking potential criminals on CCTV footage. But at least according to our unscientific little study, the field of art analysis is safe.