Behavioral characterization of dynamic facial expression perception in rhesus monkeys (Macaca mulatta) using naturalistic and synthetic stimuli

This study demonstrates that rhesus monkeys perceive facial expressions as context-dependent, functionally meaningful social signals shaped by both expression dynamics and signaler characteristics, rather than relying solely on fixed morphological categories or visual similarity.

The Gist

Imagine you are walking through a busy park. You see a dog barking, a cat hissing, and a squirrel chattering. Even without understanding the specific words they might be saying, you instantly know: "That dog is angry," "That cat is scared," and "That squirrel is just talking." You don't need a dictionary; you understand the vibe and the intent behind their faces.

This study asks a similar question about rhesus monkeys: Do they just see a bunch of moving muscles on a face, or do they actually understand what those faces mean?

Here is the story of how scientists figured it out, using some clever tricks and a lot of digital monkeys.

The Setup: A "Face-Off" Game

The researchers taught two real-life rhesus monkeys a game. They showed the monkeys videos of other monkeys making four specific faces:

1. Neutral: Just chilling.
2. Lip-Smacking: A friendly "hello" or "I'm happy."
3. Silent Bared-Teeth (SBT): A "fear grin" (showing teeth but not biting), often meaning "I'm scared" or "I submit."
4. Open-Mouth Threat: A scary "Back off!" face.

The monkeys had to press a button corresponding to the face they saw to get a juice reward. Once they got good at it, the scientists threw in a curveball: New monkeys they had never seen before.

The Big Discovery: It's Not Just About the Shape

If the monkeys were just matching shapes (like a child matching a red circle to another red circle), they would have gotten confused when the faces looked slightly different. But they didn't.

The "Aha!" Moment:
The scientists measured the monkeys' pupils. In humans and animals, pupils get bigger when we are excited, scared, or alert.

• When the monkeys saw the Threat face, their pupils dilated (got huge). They were on high alert!
• When they saw the Neutral face, their pupils stayed small. They were relaxed.

This proved that the monkeys weren't just memorizing shapes; they understood the meaning. They knew the threat face meant "Danger!" and the lip-smack meant "Friend!"

The Twist: Context is King

Here is where it gets really interesting. The monkeys didn't treat every "Fear Grin" (SBT) the same way. Sometimes they thought it was scary; other times they thought it was friendly. Why?

The scientists realized the monkeys were looking at the whole story, not just the mouth:

• Body Size: If a big, heavy monkey made the "Fear Grin," the other monkeys thought, "Hmm, that big guy isn't actually scared; he's probably just being polite."
• Eye Contact: If the monkey looked away while making the face, it changed the meaning.
• Ears and Eyebrows: Just like humans raise their eyebrows when surprised, the monkeys used their ears and eyebrows to add nuance. A "Fear Grin" with lowered eyebrows meant "I'm terrified," but with raised eyebrows, it might mean something else.

The Analogy: Think of it like a text message. If you send "I'm fine," it's neutral. But if you send "I'm fine" with a crying emoji, it means "I'm sad." If you send it with a fire emoji, it means "I'm furious." The monkeys read the "emojis" (ears, eyes, body size) to understand the real message.

The Digital Lab: Testing with Avatars

To test exactly what parts of the face mattered, the scientists created digital avatars (computer-generated monkeys). They played with these avatars like a video game:

• The "Uncanny Valley" Test: They removed the fur and color, leaving just a wireframe skeleton. The monkeys couldn't recognize the faces anymore. Lesson: You need to see a face that looks somewhat real (skin and fur) to understand the emotion.
• The "Stop-Motion" Test: They scrambled the video frames so the movement was jerky and made no sense. Surprisingly, the monkeys still understood the faces! Lesson: The specific motion isn't as important as the final shape the face makes.
• The "Human" Test: They put human faces (smiling, angry, scared) onto the monkey avatars. The monkeys were completely confused. They didn't know what to do. Lesson: Monkeys don't have a universal dictionary for all primates. They only understand their own species' "language." A human smile doesn't mean "friend" to a monkey; it's just a weird face.

The Takeaway

This study teaches us that rhesus monkeys are not just face-matching machines. They are social detectives.

They don't just look at a mouth; they look at the whole picture. They combine the mouth shape, the eye direction, the body size, and the ear position to figure out: Is this monkey a friend? Is it a threat? Is it scared?

It's like reading a book where the words are the mouth movements, but the tone of voice, the setting, and the character's size are the ears, eyes, and body. The monkeys read the whole story, not just the words.

Technical Summary

1. Problem Statement

While facial expressions are a primary channel of primate communication, the specific criteria rhesus macaques use to evaluate and categorize these expressions remain unclear. Previous studies have relied heavily on static images or match-to-sample paradigms that may introduce biases (e.g., selecting "positive" over "negative" expressions). Key unresolved questions include:

• Do monkeys categorize expressions based on low-level visual similarity (morphological features) or functional meaning (social intent)?
• Which specific facial motion components (mouth, eyebrows, ears) and contextual factors (gaze, body size) are necessary and sufficient for expression perception?
• How robust is this perception against changes in stimulus realism, motion coherence, and species-specific form?

2. Methodology

Subjects and Setup

• Subjects: Two adult male rhesus macaques (Macaca mulatta).
• Paradigm: A forced-choice categorization task. Monkeys viewed a video of a facial expression and indicated the perceived category (Neutral, Lip-smacking, Silent Bared-Teeth [SBT], or Open-Mouth Threat) by making a saccade to one of four colored targets.
• Physiological Measure: Pupil size (vertical diameter) was recorded via eye tracking to measure autonomic arousal as a proxy for emotional/physiological engagement.

Stimuli

• Part I (Naturalistic): Videos of real monkeys (training and novel test identities) and a naturalistic monkey avatar.
• Part II (Synthetic Manipulations):
  • Morphing: Continuous interpolation (25% steps) between prototypical expressions to create graded stimuli.
  • Motion Coherence: Time-scrambled frames (abolishing temporal continuity).
  • Realism Degradation: Stepwise removal of fur, color, texture (wireframe), and shape (point-light displays).
  • Species Cross-over: Human facial expressions (smile, gasp, scowl) displayed on both human and monkey avatars.

Quantification and Analysis

• Automated Coding: A novel pipeline combined DeepLabCut (deep learning-based markerless pose estimation) with the Macaque Facial Action Coding System (MaqFACS). This allowed for the objective quantification of Action Unit (AU) activation intensity (Euclidean distance from neutral).
• Statistical Modeling:
  • Multinomial Logistic Regression (MLR): To identify predictors of categorization (Expression type, Identity, Body Weight, Gaze Aversion, AU combinations).
  • Random Forest Classification: To model non-linear interactions and determine feature importance (e.g., jaw opening vs. brow lowering).
  • General Linear Models (GLM): To analyze pupil size responses.

3. Key Contributions

1. Novel Automated Quantification: The study introduces a method integrating DeepLabCut with MaqFACS to objectively quantify facial action units in dynamic videos, moving beyond subjective manual coding or static image analysis.
2. Functional vs. Morphological Decoupling: The authors demonstrate that misclassifications are not driven by visual similarity (FACS correlation) but by functional context and signaler characteristics.
3. Context-Dependent Perception: The study establishes that expression perception is not a fixed morphological category but a flexible interpretation influenced by the signaler's body weight, gaze direction, and coordinated movements of non-mouth features (eyebrows, ears).
4. Stimulus Robustness Thresholds: The research defines the minimum requirements for expression recognition: discernible facial shape and basic texture are required, but temporal motion coherence and photorealism are not.
5. Species Specificity: The findings highlight that while motion patterns can be abstracted, the specific social meaning of expressions is species-locked; human expressions do not map onto macaque categories without learned associations.

4. Key Results

Categorization and Generalization

• Monkeys successfully generalized learned categories to new real monkeys and avatars, though performance was imperfect (overall accuracy ~42% for real, ~38% for avatar).
• Threat expressions were the most accurately categorized and elicited the strongest pupil dilation (highest arousal).
• SBT (Silent Bared-Teeth) categorization was highly variable and dependent on the stimulus identity.
• Visual Similarity: Errors did not correlate with visual similarity between expressions. Instead, confusion occurred between functionally related categories (e.g., Lip-smacking vs. Neutral; SBT vs. Threat).

Influencing Factors (MLR & Random Forest)

• Body Weight: Higher body weight of the stimulus monkey significantly decreased the likelihood of an SBT being categorized as "submission" (SBT target), suggesting SBTs from large monkeys are interpreted differently (perhaps as less submissive).
• Gaze Direction: Averted gaze increased the probability of SBT categorization (avoidance signal) and decreased Threat/Lip-smack categorization (approach signals).
• Feature Integration:
  • Jaw Open: Primary predictor for Threat.
  • Teeth Bare: Primary predictor for SBT.
  • Brow/Ears: In ambiguous (morphed) stimuli, lowered brows shifted categorization from Threat to SBT or Neutral to Lip-smack. Erected ears predicted Threat.

Synthetic Stimuli Findings

• Morphing: Categorization followed a graded continuum based on expression intensity. Pupil size scaled with threat intensity.
• Motion & Realism:
  • Scrambled Frames: Recognition remained intact; temporal coherence is not required.
  • Texture/Shape: Furless and grayscale avatars were recognized. However, wireframe (no texture) and point-light (no shape) avatars failed to elicit differential categorization or pupil responses. Discernible shape and texture are necessary.
• Human Expressions: When human expressions (smile, fear, anger) were shown on avatars, monkeys showed random categorization and no differential pupil arousal. Visual similarity between human and monkey expressions did not predict choices, indicating a lack of homologous mapping without specific learning.

5. Significance

This study fundamentally shifts the understanding of primate social cognition regarding facial expressions. It argues that rhesus macaques do not perceive expressions as static, rigid morphological templates but as dynamic, context-dependent social signals.

• Evolutionary Insight: The robust detection of threats and the reliance on species-specific motion patterns suggest an innate predisposition for threat detection and a specialized neural mechanism for conspecific communication.
• Methodological Impact: The integration of DeepLabCut with FACS provides a scalable, objective tool for future ethological and neurophysiological studies of facial behavior.
• Theoretical Implication: The failure to generalize to human expressions, despite structural similarities (e.g., teeth exposure in smiles vs. SBT), challenges the notion of universal, homologous emotional expressions across primates. Instead, it supports the view that expression perception is a learned, integrated process combining visual features with social knowledge and context.

In conclusion, the paper demonstrates that facial expression perception in rhesus macaques is a sophisticated, flexible process shaped by the interplay of facial motion, signaler characteristics, and social context, rather than a simple visual matching of static features.