Facial expressions and what they really mean

If you had even dipped a toe into certain circles of Twitter last week you couldn't have missed the outrage frothing around a recent paper in Nature by Lou Safra, Coralie Chevallier, Julie Grèzes and Nicolas Baumard. The authors analysed a collection of portraits using machine-learning and argued they demonstrated an increase in expressions of "trustworthiness" over time in parallel with declining rates of interpersonal violence in society.

The pushback was immediate and dramatic from many directions. In my historian corner of Twitter there was anger that the authors had used historical sources without bothering to consult any historians. As Matuesz Fafinski of the University of Lausanne noted, the authors seem to have paid no attention to the potential bias inherent in the set of data they had chosen to use, most of which came from the National Portrait Gallery in England.

His thread is a great read and a robust defence of the science of historical research. There was also another line of criticism about the potentially terrible consequences of coding certain faces as “trustworthy”, which smacked of a darker past of trying to identify criminals and deviants by facial features.

But what fascinated me was the very complex problem at the core of the project, and which the authors’ prose presented as a simple one. What meaning do we take from facial expressions? How do we take a curve of the mouth or the position of the eyebrows and then imbue them with a whole lot of extra information that can determine whether we perceive someone as friendly or angry – in love or aloof?

What do our yellow Covid-19 signs say about us?

---

Look to lichens for dating with a difference

---

Coronavirus: The misguided search for ‘patient zero’

---

Do surgical masks guard against viral infection?

---

‘Surprising nonsense’

This problem has intrigued people for centuries. Of course Darwin himself published a treatise on the subject in 1872; On the Expression of the Emotions in Man and Animals. In the introduction he provides examples of “the surprising nonsense which has been written on the subject”. Perhaps we can add this most recent paper to that pile, but it has certainly prompted interesting discussions.

The inclusion of animals in Darwin’s book was significant and tells us much about Darwin’s approach. For other writers, expressions of emotion separated man from animals. For Darwin, they firmly tied them together. The similarity, for example, between laughing monkeys and laughing people suggested a common origin. Many of the same muscles were present in the faces of all primates, he argued, and therefore it was all the more fascinating to see how they used them differently.

But Darwin acknowledged the difficulties inherent in studying emotional expression. Movements might be “extremely slight, and of a fleeting nature”. An observer would certainly notice a change in the face, would feel a corresponding emotion, but perhaps could not pinpoint what exactly had changed.

Darwin simplified a crying infant into a series of signs: closed eyes, "contracted" forehead and "lips retracted in a peculiar manner".

This would make scientific observation difficult. And he also honed in on a problem that might have served as some warning to Safra et al. So highly tuned are we humans to reading emotions that when we expect to see an emotional expression we usually see it, even when it isn’t there.

Darwin considered and then dismissed the use of paintings and sculpture for studying emotion. He lived in the age when photography was thought to be a more “true” representation of the world but also believed artistic works had different aims. By trying to create beauty, the artist would naturally alter certain types of expressions that might not be beautiful.

Many flaws

Darwin’s study had many flaws of its own from the perspective of science today. His attempt to gauge the universality of expression across cultures (which he called races) is very clunky. He simplified expressions down to very specific, observable muscular movements.

Some of these seem hilarious now. When you last saw an indignant person, did they square their shoulders and clench their fist? He recognised that emotional expression might require cultural context but claimed, based on a rather small sample of questionnaires filled out by Europeans, that actually these expressions were universal. This suited his purpose of seeing them as another piece of evidence for evolutionary relationships.

Crying infant

The contrast between the ease with which we navigate the emotional world and the difficulty of pinpointing what it is exactly we are displaying and responding to in our expressions continues to fascinate. Darwin simplified a crying infant into a series of signs: closed eyes, “contracted” forehead and “lips retracted in a peculiar manner”. Over 100 years later scientists are still trying to convert emotions into quantifiable physical signs and the rest of are still mostly seeing a baby that needs comforting.

Dr Juliana Adelman is an assistant professor in history at Dublin City University – @AdelmanJuliana