A Multiscale Network with Supervised Contrastive Learning for Real-Time Facial Emotion Recognition

This paper presents a deep learning-based system utilizing a multiscale network and supervised contrastive learning to achieve real-time facial emotion recognition by modeling continuous expression changes, demonstrating satisfactory performance on standard datasets for applications such as psychological counseling.

The Gist

Imagine you are trying to understand how a friend is feeling just by watching them. Sometimes, a smile means they are happy; other times, it might mean they are being polite or hiding sadness. Now, imagine trying to do this with a computer, but the computer only gets a single snapshot in time. It's like trying to guess the plot of a movie by looking at just one frame—it's easy to get it wrong.

This paper introduces a new system called MSFERNet (Multi-Scale Facial Emotion Recognition Network) designed to solve this problem. Think of it as a "smart camera" that doesn't just look at a face once, but watches how the face changes over time, much like a psychologist watching a patient during a session.

Here is a breakdown of how it works, using simple analogies:

1. The Problem: Emotions are a Movie, Not a Photo

The authors point out that emotions aren't static; they flow and change. A person might start neutral, get slightly annoyed, and then calm down. Most old computer systems are like photographers who take a single picture and guess the mood. This paper argues that to really understand someone, you need to watch the "movie" of their face.

2. The Solution: A Multi-Lens Camera (MSFERNet)

The core of their system is a new type of AI architecture they built. Imagine a detective trying to solve a case.

• The "Wide-Angle" Lens: Some parts of the system look at the big picture (the overall shape of the face).
• The "Zoom" Lens: Other parts zoom in on tiny details (the twitch of a lip or a wrinkle in the brow).
• The "Memory" (Residual Learning): Just like a detective who remembers clues from earlier in the day, this system uses "residual blocks" to remember what it saw previously so it doesn't lose track of the story as it digs deeper.
• The "Spotlight" (Attention Mechanism): The system has a built-in spotlight (called CBAM) that ignores the background (like a messy room or a window) and focuses strictly on the face, highlighting the most important parts.

3. Training the Brain: Learning from Groups

To teach this system, the researchers didn't just show it pictures and say "This is happy." They used a technique called Supervised Contrastive Learning.

• The Analogy: Imagine a teacher showing a student a pile of red apples and a pile of green apples. Instead of just saying "Red is red," the teacher says, "Look at how similar these red apples are to each other, and how different they are from the green ones."
• By grouping similar emotions together and pushing different emotions apart in its "mind," the computer learns a much clearer picture of what each emotion actually looks like.

4. Simplifying the Language: The Three-Color System

The researchers realized that real life is complicated. A standard dataset has 7 or 8 different emotions (Angry, Disgust, Fear, Sad, Happy, Surprise, Neutral, etc.).

• The Analogy: They decided to simplify this into a "Traffic Light" system for their real-time application:
  • Green: Positive (Happy)
  • Yellow: Neutral
  • Red: Negative (Angry, Disgust, Fear, Sad)
• They purposely left out "Surprise" because, like a plot twist in a movie, it can mean anything depending on the context, making it too confusing for a quick analysis.

5. The Real-Time Tool (RT-FER)

They built a user-friendly application called RT-FER.

• How it works: You can upload a video or use your webcam. The system grabs your face from every frame, runs it through the "Multi-Lens Camera," and gives you a score.
• The Score: It translates the emotion into a number between -1 and 1.
  • -1 is pure negative.
  • 0 is neutral.
  • +1 is pure positive.
• The Graph: As the video plays, the system draws a line graph showing how your mood "rides the waves" up and down over time.

6. The Results: Fast, Light, and Accurate

The team tested their system on standard datasets (like FER13 and CK+).

• Performance: It did very well, getting about 96.77% accuracy on one dataset and 81.08% on their simplified 3-emotion version.
• Efficiency: The best part is that the system is "lightweight." It only has 2.37 million parameters (think of these as the number of rules the computer has to memorize). Compared to other systems that are like heavy, slow trucks, this one is like a nimble bicycle. It's small enough to run on regular devices without needing a supercomputer.

7. The Catch (Error Analysis)

The authors were honest about the flaws. If the training data has "bad photos"—like a picture with a logo instead of a face, or a face covered by a giant watermark—the system gets confused. It's like trying to teach a child to recognize dogs using pictures of cats with dog ears drawn on them.

Summary

In short, this paper presents a smart, lightweight AI that watches faces like a human observer, looking for changes over time rather than just a single snapshot. It simplifies complex emotions into a clear "Positive/Negative/Neutral" score, making it a useful tool for tracking emotional shifts in real-time videos.

Technical Summary

Technical Summary: A Multiscale Network with Supervised Contrastive Learning for Real-Time Facial Emotion Recognition

Problem Statement

Real-time facial emotion recognition (FER) presents significant challenges, particularly in video-based scenarios where emotional states evolve continuously rather than discretely. A primary difficulty lies in the high inter-subject variability of facial expressions and the ambiguity of emotions (e.g., a smile may indicate happiness, politeness, or sarcasm depending on context). Furthermore, existing research has largely focused on static image recognition or single-frame classification, leaving a gap in the ability to analyze and monitor emotional changes over extended timeframes. This limitation hinders the comprehensive understanding of an individual's psychological state, which is crucial for applications in psychology and counseling where the ratio of experts to patients is insufficient.

Methodology

The authors propose a two-phase system comprising a deep learning architecture for feature extraction and classification, and a real-time application interface.

1. MSFERNet Architecture

The core of the system is MSFERNet (Multi-Scale Facial Expression Recognition Network), designed to address feature degradation and vanishing gradients common in deep sequential CNNs. The architecture incorporates:

• Backbone: It utilizes the early stages of a pre-trained EfficientNet-B0 to extract low-level and mid-level semantic features, reducing computational complexity compared to using the full network.
• Residual Refinement: Extracted feature maps pass through a refinement block containing a $3 \times 3$ convolution, Batch Normalization, ReLU, and a Residual Block with skip connections to preserve identity mappings and stabilize gradient flow.
• Multi-Scale Feature Extraction: The network employs parallel convolutional branches with $3 \times 3$ and $5 \times 5$ kernels.
  • Stage 1: Branches are combined via element-wise addition.
  • Stage 2: Branches are concatenated channel-wise to preserve complementary information from different receptive fields.
• Attention Mechanism: A Convolutional Block Attention Module (CBAM) is applied after each multi-scale stage to sequentially emphasize informative facial regions (channel and spatial attention) while suppressing background noise.
• Classification Head: Features are downsampled, globally pooled, and passed through fully connected layers (128 and 64 units) with dropout (0.3) to prevent overfitting.
• Supervised Contrastive Learning: A projection head maps features into a normalized embedding space. The model is trained using a combined loss function:
  $$L = 1.0 \times L_{cross} + 0.1 \times L_{sup}$$
  Where $L_{cross}$ is the Categorical Cross-Entropy Loss and $L_{sup}$ is the Supervised Contrastive Loss, designed to learn better representations of emotional features by pulling positive samples (same class) closer and pushing negative samples apart in the embedding space.

2. Dataset Preprocessing and Modification

The study utilizes the FER13 and CK+ datasets. To align with the goal of aiding psychologists in identifying broad mental states, the authors modified the standard 7-class FER13 dataset into a 3-class system:

• Positive: Derived from the 'Happy' class.
• Negative: Merged from 'Angry', 'Disgust', 'Fear', and 'Sad'.
• Neutral: Retained as is.
• Note: The 'Surprise' class was excluded due to its high contextual dependency and tendency to evoke mixed emotions.
• Preprocessing: Images were resized to $128 \times 128$, and standard augmentations (shifting, zooming, shearing, flipping) were applied. Corrupted images were filtered out.

3. RT-FER System

A user-friendly application named RT-FER was developed to demonstrate real-time monitoring. It captures live video or processes uploaded videos, extracts faces from frames, and feeds them to the trained MSFERNet. The system outputs:

• Emotion Prediction: The predicted class with confidence scores.
• Emotion Scoring: A continuous score calculated as $Score = p_{positive} - p_{negative}$ (mapping Negative to -1, Neutral to 0, Positive to 1).
• Visualization: A graphical interface displays the video feed alongside a real-time plot tracking the emotion score over time.

Key Contributions

1. MSFERNet Architecture: Proposal of a multi-scale, attention-based network that integrates transfer learning, residual mechanisms, and supervised contrastive learning.
2. Dataset Adaptation: Creation of a modified 3-class FER13 dataset tailored for psychological state analysis, addressing the lack of standard datasets for broad emotional categories.
3. RT-FER Application: Development of a functional GUI that allows for real-time emotion monitoring and the visualization of emotional changes over time, including a video player to observe context-induced emotional shifts.

Experimental Results

The model was evaluated on FER13 (original 7-class and modified 3-class) and CK+ datasets using an 80:10 train-test split.

• Performance:
  • FER13 (7-class): 66.73% accuracy.
  • FER13 (3-class): 81.08% accuracy.
  • CK+: 96.77% accuracy.
• Efficiency: The model contains only 2.37 million trainable parameters, making it significantly more resource-efficient than state-of-the-art models like AlexNet (62.30M) or VGGNet (84.00M).
• Impact of Supervised Contrastive Loss: The inclusion of $L_{sup}$ improved accuracy across all datasets (e.g., FER13 7-class improved from 64.19% to 66.73%; CK+ improved from 95.56% to 96.77%).
• Comparison: The proposed MSFERNet outperformed several existing SOTA models on both FER13 and CK+ datasets while maintaining a lower parameter count.

Significance and Limitations

The paper claims that the proposed system bridges the gap between static emotion recognition and continuous psychological state monitoring. By providing a tool to track emotional changes over time, it offers a potential aid for psychologists to gain additional insights into a subject's emotional state, potentially alleviating the burden of manual observation.

The authors modestly acknowledge limitations, noting that despite preprocessing, the training data contained erroneous samples (e.g., images with logos or watermarks) which impacted training. They also highlight that real-time recognition remains challenging due to variations in image quality and the inherent ambiguity of facial expressions. The work concludes that while the current results are satisfactory, future improvements could be achieved through training on larger real-world datasets and incorporating stronger attention mechanisms.