Support Hyperallergic’s independent arts journalism.
Art historians and critics have spent countless hours analyzing the emotions of famous portrait subjects, from the mystery of Mona Lisa’s smile and the anguish of Picasso’s “Weeping Woman” to the tortured angst in Munch’s “The Scream.” This process of reading feelings in faces has long been considered an intuitive rather than scientific one — but perhaps artificial intelligence could soon help us settle these questions once and for all.
Microsoft’s Project Oxford photo research division has just released a new demo of a tool that detects emotions in photographed faces using machine-learning techniques. When you upload a picture to the site, the software scans the subject’s face and attempts to read the feelings behind their expression. It then presents a scorecard for each subject, ranking them on eight emotions: anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise.
The site does offer a disclaimer: “Recognition is experimental, and not always accurate.” Though they were developed after years of research, these robots are still learning, and their emotional vocabulary is pretty limited. Still, could this app offer any new insights into the emotional lives of famous portrait subjects? To test it out, we ran five artworks (admittedly not all photographs) through the app. It was a weird exercise in assigning numerical values to things usually considered unquantifiable. Here are some feelings in art from a robot’s perspective; see how its analysis compares to your own real, live human impressions.
Leonardo da Vinci, Mona Lisa (c. 1503)
This isn’t the first time the Mona Lisa has been subject to the scrutiny of emotion-recognition technology: back in 2005, computers at the University of Amsterdam confirmed longstanding human suspicions that this most famous of sitters is “mainly happy.” Microsoft’s new app agrees — the only categories in which the Mona Lisa received ratings higher than zero were “happiness” (0.43) and “neutral” (0.55). Before the advent of such advanced technology, even emotionally illiterate viewers could come to a similar conclusion by reading the painting’s Italian title, “La Gioconda,” which, in addition to being a play on the presumed sitter’s name (Lisa del Giocondo), literally translates to “jocund,” or “happy.”
Richard Avedon, “Karl Rove,” from Portraits of Power (2004)
Former White House Deputy Chief of Staff Karl Rove, in a photo from Richard Avedon’s series Portraits of Power, was rated the happiest of any subject we ran through the app: his smug, self-satisfied smirk earned him a nearly 100% happiness score. Microsoft’s technology failed to detect any of the politician’s warmongering bile. But Rove himself wasn’t very happy when he saw this portrait; he said it “makes me look like a complete idiot” and accused Avedon of being “an elitist snob who deliberately set me up.”
Dorothea Lange, “Migrant Mother” (1936)
Microsoft’s robot did manage to pick up on the sadness in the face of Florence Thompson, a “destitute pea-picker” and 32-year-old mother of seven in Depression-era California, shot by Dorothea Lange in 1936. But the unfeeling AI suggests that this vast melancholy can be quantified, giving it a neat little score of 0.13.
Cindy Sherman, “Untitled #414” (2003)
What emotion could Cindy Sherman’s super creepy, painted-on grin possibly signify? Happiness, says Microsoft’s still-learning robot. The only other subject it rated happier than this deranged evil clown was Karl Rove. Go figure.
Johannes Vermeer, “Girl with a Pearl Earring” (c. 1665)
Vermeer’s “Girl with a Pearl Earring” is often called “The Mona Lisa of the North” and described as enigmatic and mysterious. “The more you look at her the more the expression changes,” Melissa Buron, a curator at the De Young Museum, once said of the painting’s subject. “At first she seems caught in a moment, startled. And then she seems to be responding to a question.” Microsoft’s app doesn’t quite get that specific, assigning “Girl” 85% neutrality and 13% happiness, with tiny traces of sadness, surprise, fear, contempt, and anger lurking beneath.
There are plenty of fields in which emotion-detecting software like this might have practical uses. Marketers could use it to gauge people’s reactions to products, Microsoft’s Allison Linn writes, or to create consumer tools, such as a messaging app, that offers up “different options based on what emotion it recognizes in a photo.” And creepily, in our surveillance state, security professionals could use this tech to help keep an eye on angry or contemptuous-looking potential criminals on CCTV footage. But at least according to our unscientific little study, the field of art analysis is safe.
“Black infants in America are now more than twice as likely to die as white infants—11.3 per 1,000 black babies, compared with 4.9 per 1,000 white babies, according to the most recent government data—a racial disparity that is actually wider than in 1850, 15 years before the end of slavery, when most black women were…
he ownership of images has a long and nuanced legal history, which has evolved dramatically in recent years as cultural standards and photographic technologies have rapidly advanced
The show, which honors the 50th anniversary of an exhibition history once ignored, continues a series of projects documenting Wilmington’s contemporary art scene.
Renty and his daughter Delia. Renty was an enslaved African, kidnapped from the Congo, sold and forced into slave labor on the South Carolina plantation of B.F. Taylor
What is the relation between possessing a person, possessing their image, and dispossessing their progeny
As a scholar of African American history and photography whose work has focused on the status of violent images in museums and archives, I fully support the validity of Ms. Tamara Lanier’s claim and the amicus brief.
Two K-12 art teachers will each receive a $1,000 cash gift and an additional $500 to put toward classroom art supplies. Nominations are due October 31.
The daguerreotypes of Renty Taylor, Delia, Drana, Alfred, Jack, George Fassena, and Jem remained in an unused storage cabinet until 1975, when it was discovered by an employee of the Peabody Museum.
I am writing in support of the amicus curiae brief submitted by Professor Ariella Aïsha Azoulay of Brown University for the full restitution of the daguerreotypes of Renty Taylor and his daughter Delia, currently held by Harvard University, to their familial descendant, Tamara Lanier.
We cannot be indifferent to the long-lasting effects of photography. The photographs at the center of Lanier v. Harvard are relentless in making Renty and Delia Taylor work and perform as slaves. The pain inflicted on them has not ceased. Photography has the capacity to propagate harm, and we have the moral obligation to interrupt…