Portraiture is an irresistible genre of painting. As representations of people who lived hundreds, even thousands, of years ago, they seem to afford direct access to what can often feel like an inaccessible past, humanizing people who inhabited cultures far removed from our own. Portraits are, to use a phrase common among our students, relatable.
So, it is unsurprising to see the regularity with which news outlets and social media report with breathless excitement scientific discoveries about the past that emerge from the application of the latest imaging technologies, artificial intelligence software, and medical diagnostics of very old portraiture. Portraiture attracts the attention of those who work in the sciences precisely because, with its often lifelike appearance, it seems to offer itself up as evidence, providing faces that might be psychoanalyzed or diagnosed. For example, a study published in the Annals of Human Biology, asserts a correlation between inbreeding and the outsize jaws represented in portraits of the Hapsburgs; a paper published in Nature Communications argues that a rise in displays of trustworthiness between the years 1500 and 2000 can be ascertained from paintings hung in the National Portrait Gallery of London; and a CNN article claims that Leonardo da Vinci’s depictions prove he had an eye condition partly responsible for his artistic genius.
But what kind of evidence is portraiture? Many of these studies presume an equivalence between paintings and the people they represent. In other words, their assumption is that the features of painted faces are the facts of the flesh-and-blood countenances to which they refer. This assumption is not only false; it is preposterous. A portrait, like any other kind of art, is the product of intersecting demands and desires. It is governed by the goals of the patron, the preferences of the artist, representational norms of a given time and place, the expectations of the anticipated audience, and its intended location and function. None of these predicates can be stripped away or controlled for to reveal the “true,” real-life face.
Take the Habsburg jaw article: In lay terms, the authors looked at a lot of paintings representing members of the Habsburg family and determined that any jaws that were large in relation to the face were the result of inbreeding. The baseline empirical problems present themselves at the outset: the 66 paintings examined were produced over more than 100 years by different artists working in different regions and styles. The minute measurements required to even “diagnose” a condition from them cannot be meaningfully compared across such a diverse body of evidence. But there is a high-level issue, one immediately recognizable to any art historian and almost immediately disqualifying: The genre of royal or dynastic portraiture has as one of its fundamental tasks the establishment of a lineage and legitimacy. One way in which an artist could have signaled a sitter’s pedigree would have been to portray him or her with a particular physical feature for which a relative or ancestor was known. In other words, the painted portrait constituted a key piece of evidence, not of the subject’s actual face, but of the subject’s legitimacy.
Once a particular trait — say, that jaw — entered the canon of Habsburg portraiture, it was in the sitter’s interest to have an artist paint him or her with it to signal their membership in the dynasty. In a lecture delivered over 20 years ago on these very portraits, David Davies remarked that, critical to the Catholic dynasty that ruled over enormous and diverse territories was not only “purity of faith but also purity of blood. Together with legitimate birth, these were to be the firmaments of the code of honour with that which society was so deeply imbued.” What better way to “prove” one’s purebred status as a Habsburg than to have a portrait painted with the trademark trait? According to some 17th-century accounts, portraits of this family were vetted for their resemblance to the individuals they represented, and even censored when they were considered to lack the requisite likeness. But what art history teaches us is that the very concept of “likeness” itself differs across time and space. Our idea of an “accurate” representation of a face differs from the ideas about accuracy in other periods and places. “Accuracy” could even encompass squaring a sense of a sitter’s moral character with how their face should look in a painting.
Changing ideas about accuracy relate to an even deeper problem with these supposedly scientific approaches: The researchers are unaware that both portraiture and the ideas portraits express have a history. For one, the paintings that serve the diagnostic impulses of scientists are unanimously, and necessarily, naturalistic and European in origin. Naturalism is a term that art historians use to refer to something that has the appearance of reality. But naturalism is a fashion: Certain groups of people preferred naturalistic paintings more at certain points in time, and its popularity waned in others. Relying solely on portraiture that conforms to a presentist and Western-biased idea of naturalism privileges a narrow body of evidence.
Similarly, the writers of these studies assume that the facial expressions in these portraits have a timeless meaning. For instance, a recent paper that argues that “trustworthiness displays” in portraiture increased in accordance with a “rise of democratic values observed in Western Europe.” The authors quantify trustworthiness by measuring the curve of lips and width of eyes. But what, exactly, is “trustworthiness” to begin with? This is a term that the authors of the article leave undefined, and, as those who work in the field of intellectual history would argue, the concept of trust and how it is expressed are by no means universal. As Sarah E. Bond and Nyasha Junior’s recent Hyperallergic article on the “New Jim Code” further makes clear, the study’s dataset drew in large part from a corpus of portraits depicting white, elite men, a body of evidence that should not be treated as representative of Europe as a whole. To flatten the diverse evidence of half a millennium of painting in order to quantify deviations in a few basic parameters is not just bad art history; it is bad science.
The imperative to broaden one’s dataset should not be taken lightly. The many painted portraits produced at the royal court of the Mughals, a Muslim dynasty of Turco-Mongol origins that ruled over much of South Asia between the 16th and 19th centuries, are a case in point. Many have long regarded the highly detailed 17th-century portraits of Emperor Jahangir (who ruled from 1605 to 1627), for example, as representative of reality. Yet these images, while naturalistic in appearance, are nevertheless highly idealized. Rather than depicting the emperor exactly as he appeared, they portray him according to a specific cultural paradigm that favored some physical features over others. One wonders, furthermore, what facial recognition algorithms, which are designed to analyze front-facing images, would make of Mughal portraiture. After all, the strict side profile format was the preferred mode of depiction within the Mughal empire for well over 200 years.
Treating paintings of Mughal women as objective portrayals is an equally fraught enterprise. A painting in the Cleveland Museum of Art’s collection that depicts a woman holding a portrait of Jahangir may very well represent the emperor’s favorite wife, Nur Jahan, but it is unlikely that the artist who created the work ever saw the Mughal queen in person. Women of the royal household were generally inaccessible to the public; the only men who would have had visual access to Nur Jahan, for example, were the emperor and other male members of her family. Thus, the painter of the Cleveland double portrait had to portray Nur Jahan according to conventions and ideals of female beauty, as opposed to drawing upon direct observation. This phenomenon of course helps to explain why portraits of Mughal women look so very similar to one another.
Portraiture, and representation more broadly conceived, has long been — and remains— a central concern of the discipline of art history, and yet many of these scientific studies simply fail to engage with this scholarship. It may be that scientists perceive art historical work to be overly qualitative and thus insufficiently rigorous. Setting aside the fallacy that non-quantitative research is inherently imprecise, it is simply untrue that art historians dispense with data wholesale. Quite the contrary, art historians analyze data all the time, and many even employ quantitative methods in their research of cultural artifacts and the built environment.
Works of art and architecture are among the data that art historians attend to, but data can also take the form of archives, inventories, chronicles, poetry, and epigraphs, to name but a few examples. The data about these data, also known as metadata, constitute yet another rich field of art historical inquiry. The metadata of painted portraiture, for example, include such physical properties as dimensions and materials, but they also encompass the dates, conditions, patrons, makers, and locations associated with their production, use, and circulation. While many art historians today use digital tools and methods to analyze and interpret their datasets, art history’s status as a data-driven discipline long predates the recent computational turn.
There are, however, critical differences between how art historians and scientists treat the data and metadata of art, including that which pertains to painted portraiture. Namely, where the authors of these scientific studies consider the painted data as correlating with the reality of the subjects represented, the metadata as objective tools of classification, and the source of their painted specimens (museums) as effectively random, historians of art understand that paintings, the language used to classify them, and the museums that house and display them are all themselves interpretative mechanisms and the products of subjective, human intellection.
Two notorious figures offer object lessons in the perils of denying the subjective nature of these data. Johann Winckelmann was an 18th-century art historian who proclaimed ancient Greek sculpture to be the epitome of artistic perfection. Taking this opinion — which he presented as fact — a step further, he claimed that such sculpture matched perfectly the appearances of the race of people who made them, a race of people he positioned as the pinnacle of humanity. Blending art and biology, writings such as Winckelmann’s fueled the colonial enterprise that depended on a hierarchy of races for its rationale.
With the advent of the photographic portrait, European intellectuals took these pseudo-scientific endeavors further, leveraging this latest technology to lend credibility to the emerging field of psychology. The polymathic eugenicist Francis Galton, for example, believed that criminality manifested in the face. Associating facial features with behavioral tendencies, he invented “pictorial statistics” to fabricate composite portraits of the “criminal type.” Using software to seek out signs of a valued personality trait in paintings may not exactly be phrenology, but its intellectual forebears are. Operating with complete ignorance of this history is just one area where such scientific studies can go horribly wrong.
Another danger lies in the treatment of algorithms as free of human bias. The recent study of “trustworthiness displays,” for example, analyzed European portraiture using OpenFace, an open-source facial recognition program that was engineered using a dataset of overwhelmingly white subjects, and five sets of avatars modeled on Caucasian physiognomy. Yet the authors failed to acknowledge in explicit terms the biases that are embedded in these models. Also concerning is the fact that police around the world are increasingly deploying biased software like OpenFace to surveil their own citizens, the result being that the most vulnerable populations are often the most frequently targeted. As work by Ruha Benjamin and Safiya Umoja Noble shows, algorithms reproduce and reinscribe racial, ethnic, and other prejudices precisely because they are human-made. Algorithms, in other words, do not write — nor will they fix — themselves. It is incumbent upon scholars, scientists, and programmers to recognize their own partialities. As recently discussed by Bond and Junior, the developers of Beauty.AI, a beauty contest evaluated by artificial intelligence, created an algorithm that favored entries with lighter skin. It was only after they had come to terms with the racial and other biases embedded in their original algorithm that they could begin to explore ways to create more inclusive code and draw upon more diverse datasets.
Applications that use facial recognition are a fact of 21st-century life, poised to become ubiquitous in public and even private domains. What is so worrying about these scientistic studies of portraiture is not simply that they distort our understanding of history — often in the service of facile, white supremacist arguments about the so-called triumphs of Western civilization — and exhibit total ignorance of and disdain for the humanistic disciplines. Just as alarming is the message they send both to the casual reader of the news reports that hype them and to the students whom their authors teach: that representation is reality. The designers of surveillance technology are being trained in this dangerous school of thought, one which popular media has already normalized for the consumers of this same technology.
