One of the most extraordinary aspects of human life is the ability to express emotions. As a social species, we have developed rich capacities to interact with each other. From our earliest age, social relationships dominate our lives and contribute to making us who we are. Ingrained with the need for social connection, emotions help us to communicate with others, judge their intentions and navigate our behavioural responses. What seems like an effortless skill involves a range of complex and relational processes. A particularly intricate aspect of human communication is its dynamic nature: rather than being like a single snapshot, emotional expressions are fleeting — we cannot press a ‘Pause’ button to halt another’s emotional expression, or indeed our own.

Unfortunately, this temporal quality of facial displays has been frequently overlooked in emotion research, most of which has relied on the use of still images or photographs. Such stimuli often consist of actors portraying high-intensity emotions based on pre-defined and stereotypical patterns of facial actions, commonly including the basic six emotions: happiness, anger, fear, sadness, disgust and surprise. While these static prototypes are well recognised due to their simplified nature, they hardly ever occur in real life — this approach is about what the person is perceived to be expressing, rather than what we think the person is really feeling at that moment.

The latter aspect is particularly intriguing. Is it possible to recognise an expression and yet have the impression that the person is feeling something different? Take the smile: the curved line of a mouth is universally perceived as a happy expression. Yet we all know that not all smiles are happy. In an attempt to distinguish between ‘genuine’ and ‘false’ smiles, work has predominantly focused on morphological features of the face. The most well-known of these is the Duchenne marker, indicated by a crinkling of the skin around the eyes and is said to accompany or constitute a genuine smile. When presented with a static image of such a smile, we tend to perceive it as expressing a felt positive emotion. However, could this impression change when dynamic information is added?

An example from Hollywood might prove valuable. At the Oscars ceremony in 2003, one of the favourites for the award of best director was Martin Scorsese for his epic drama Gangs of New York. But when Roman Polanski was announced as the winner of the award, Scorsese’s response was that of great disappointment, before he realised the camera was back on him and he immediately resumed a smile [hover to see gif]. One could debate whether his smile is a Duchenne smile; it seems improbable that happiness would have been high on his list of priorities! And the speed with which his ‘smile’ unfolds simply looks too quick to reflect genuine enjoyment of his rival’s success. This raises the following question: does dynamic information contribute to the interpretation of an emotional expression?

Determined to find out, I created video clips of smile expressions that systematically differed in their onset, apex and offset duration. When I showed them to people and asked them to rate how genuine they thought the smiles were, their ratings systematically varied with the ‘dynamic trajectory’ of the smiles. Specifically, smiles that unfolded quickly (that is, those with a short onset) and disappeared abruptly (those with a short offset) made the target’s smile appear less authentic and less believable. This also led people to assign lower ratings to the target’s dispositions, such as their trustworthiness, attractiveness and flirtatiousness.

Two conclusions can be drawn from these findings. First, people try to infer meaning beyond what is stereotypically recognised or thought of as an emotion. Second, the dynamic quality of facial expressions provides a source of valuable information that is used in the course of its display.

Facial motion promises to remain a topic of scientific interest for the foreseeable future. With computing entering the social domain, it is possible to communicate not only with other human beings, but increasingly artificial entities. These can be computer-animated characters, robots, or even virtual agents. Whether embodied or digital, their appearance and life-like demeanour is becoming more and more realistic. To gain users’ acceptance, their portrayal of emotions are an essential ingredient in revealing a character’s personality and influencing the quality of an interaction. To achieve this, we must establish how emotions should be expressed so that an artificial entity appears convincing and authentic.

From previous research we know that people’s affinity to artificial entities does not increase in a linear way with the degree of human-likeness. In fact, such characters can be subject to the so-called ‘uncanny valley’ effect when their appearance falls short of emulating that of actual human beings. Classic examples of this can be found in CGI-heavy animated films such as The Polar Express or The Final Fantasy: The Spirits Within, which many viewers find disturbing and off-putting, particularly when movement is involved. The way in which artificial systems emit or produce socially-relevant information will be an issue of continued interest if we are to create emotionally appealing and usable systems, where the temporal features of facial expressions can provide a great insight. Understanding the way we express ourselves and interpret each other as humans is crucial to making this a success.