💻 computer science

End-to-End Facial Expression Detection in Long Videos

This paper introduces FEDN, an end-to-end Facial Expression Detection Network that unifies expression spotting and recognition through multi-scale temporal attention mechanisms, achieving state-of-the-art performance on public benchmarks while also revealing a significant discrepancy between expert-annotated and self-reported emotion labels.

Original authors: Yini Fang, Alec F. Diallo, Frederic Jumelle, Bertram Shi

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yini Fang, Alec F. Diallo, Frederic Jumelle, Bertram Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a long movie and trying to understand the characters' feelings. Usually, computers do this in two separate steps: first, they try to figure out when a character starts and stops smiling or frowning (like a timer), and second, they try to guess what that smile or frown means (like a translator).

The problem is that these two steps have been done separately, like having one person watch the clock and another person guess the emotion, without them ever talking to each other. This paper introduces a new system called FEDN that does both jobs at the same time, like a single detective who watches the clock and guesses the emotion simultaneously.

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "Split-Brain" Approach

Before this paper, computers treated spotting an emotion and recognizing it as two different tasks.

Spotting: "When did the smile start and stop?"
Recognizing: "Is that a happy smile or a sarcastic one?"
Doing them separately is like trying to assemble a puzzle while blindfolded, then taking off the blindfold to see if you got the picture right. The paper argues that knowing what the emotion is helps you know when it started and stopped, and vice versa.

2. The Solution: FEDN (The "All-in-One" Detective)

The authors built a new network called FEDN that unifies these tasks. Instead of two separate workers, it's one smart worker that learns from both tasks at once.

How it sees time (The "Zoom Lens" Analogy):
Facial expressions are tricky. Some are quick blinks of emotion (like a flash of anger), while others are long, slow shifts (like a lingering sadness).

The Segment Attention: Imagine looking at a single sentence in a book. This module zooms in on tiny, quick movements within a short clip of the video to catch those fast, subtle changes.
The Sliding Window Attention: Imagine reading a whole paragraph to understand the context. This module looks at a broader stretch of time to understand the "story" of the emotion.
The Pyramid: The system combines these "close-up" and "wide-angle" views together, like a photographer using different lenses to make sure they don't miss anything, whether the emotion is short or long.

3. The Big Discovery: The "Two Truths" Problem

One of the most interesting findings in the paper isn't about the code, but about the data itself. The researchers looked at a dataset called CAS(ME)2 and found a surprising mismatch:

Expert Labels: Third-party experts watched the videos and labeled the emotions based on what they saw on the face.
Self-Reported Labels: The people in the videos later told the researchers what they felt inside.

The Analogy: Imagine an actor crying on stage. The expert watching says, "That is sadness." But the actor later says, "I was actually feeling 'helpless' or 'sympathy'."
The paper found that these two labels often disagreed. Sometimes the experts labeled a feeling as "others" or "surprise," while the person actually felt "happiness." This suggests that our current "ground truth" (the correct answer key) might be flawed because it relies on guessing what someone feels just by looking at their face, which isn't always accurate.

4. The Results: Faster and Smarter

The new FEDN system was tested on three different video datasets.

Better Accuracy: It beat all the previous "split-brain" models. It was better at finding the exact start and stop times of emotions and better at naming the emotion correctly.
Efficiency: It didn't need heavy, slow-motion tracking tools (like optical flow) that other systems used. It was lightweight and fast, like a sports car that gets great gas mileage.
Joint Learning: When the system was allowed to learn spotting and recognition together, it got even better at spotting than when it was forced to learn them separately. It's like a student who learns math and physics together understanding both subjects better than if they studied them in isolation.

Summary

In short, this paper says: "Stop treating 'when' and 'what' as separate problems. Let the computer learn them together using a smart, multi-level attention system. Also, be careful with the answer keys we use to train these computers, because what a person says they feel doesn't always match what an expert thinks they see."

Technical Summary: End-to-End Facial Expression Detection in Long Videos

Problem Statement
Facial expression analysis in long videos is traditionally divided into two distinct tasks: spotting (identifying the precise onset and offset temporal boundaries of an expression) and recognition (classifying the emotional category within those boundaries). Despite their interdependence, existing approaches treat these tasks in isolation. Spotting models are optimized for fine-grained temporal localization, while recognition models assume pre-segmented clips. This separation leads to suboptimal performance because accurate recognition relies on reliable temporal boundaries, and semantic understanding of an expression can help resolve ambiguous onsets and offsets. Furthermore, existing detection frameworks often rely on computationally expensive 3D CNNs or optical flow, which struggle with the subtle, transient, and sparse temporal signals characteristic of facial expressions.

Methodology: FEDN
The authors propose FEDN (Facial Expression Detection Network), the first model to unify spotting and recognition into a single, parallel, end-to-end detection task. The architecture consists of three primary components:

Backbone:
- Instead of 3D CNNs, FEDN employs a space-time separable architecture using a pretrained 2D facial feature extractor (e.g., ResNet18 on AffectNet) to encode aligned face frames.
- To handle long videos efficiently, a sliding window mechanism is used. Each window is divided into overlapping segments.
- Two novel temporal attention modules are introduced:
  - Segment-level Attention: Captures fine-grained local dynamics by aggregating frames within each segment using multi-head attention and average pooling.
  - Sliding Window Attention: Captures broader temporal context by applying attention across the sequence of segments within the sliding window.
- These modules allow the model to learn shared representations that leverage information from both tasks simultaneously.
Neck:
- A multi-scale temporal feature pyramid is constructed to accommodate expressions of varying durations.
- The backbone output is processed through 1D convolutional layers with stride 2 to generate feature sequences at four different resolution levels ( $s/4, s/8, s/16, s/32$ ).
Head:
- The head predicts expression intervals (start/end positions), confidence scores, and emotion labels at each temporal scale.
- It utilizes a hybrid prediction strategy: one anchor-free interval (directly regressing start/end locations) and five anchor-based intervals (using predefined templates based on dataset statistics).
- Non-Maximum Suppression (NMS) is applied at the video level to consolidate overlapping predictions.

Learning Objectives
The model is trained with a joint loss function combining:

Interval Loss ( $L_b$ ): 1D Distance IoU (DIoU) loss for positive intervals to align predicted boundaries with ground truth.
Recognition Loss ( $L_c$ ): Binary Cross-Entropy (BCE) loss for emotion classification.
The total loss is a weighted sum of these components, allowing for joint optimization where emotion labels guide the attention mechanisms to focus on semantically meaningful frames.

Key Contributions

Unified Framework: FEDN is the first model to treat facial expression spotting and recognition as parallel tasks within a single end-to-end detection network, enabling joint feature learning and co-adaptation.
Novel Evaluation Metrics: The authors perform the first evaluation of facial expression analysis using detection metrics (specifically Mean Average Precision, mAP) that simultaneously assess temporal localization and emotion classification accuracy, moving beyond the separate evaluation of spotting and recognition.
Efficient Architecture: The introduction of segment-level and sliding window attention modules captures both short- and long-term facial dynamics without the computational cost of 3D CNNs or optical flow.
Benchmarking Insights: The study reveals a systematic discrepancy between expert-annotated and self-reported emotion labels in the CAS(ME)2 dataset, highlighting a challenge in current ground-truth validity.

Results

Performance: FEDN achieves state-of-the-art results on three public benchmarks (CAS(ME)2, CAS(ME)3, and SAMM-Long). On CAS(ME)2, it improves the detection mAP by 3.9-fold compared to the STR baseline (from 13.4 to 51.6) and achieves the highest spotting and recognition F1 scores.
Efficiency: FEDN outperforms I3D-based backbones with fewer parameters (24.98M vs. 30.52M) and without requiring optical flow.
Ablation: Removing the attention modules significantly degrades performance (dropping mAP from 51.6 to ~38-47), confirming their necessity. Joint training with recognition labels improves spotting accuracy compared to training with spotting labels alone.
Label Discrepancy: Experiments show that models trained on self-reported labels often achieve higher precision at strict IoU thresholds, suggesting self-reports may better reflect internal emotional states, whereas expert annotations may introduce noise or overextended boundaries.

Significance
The paper argues that unifying spotting and recognition allows for a more realistic assessment of how well a model identifies and interprets expressions in continuous videos. By demonstrating that shared representations improve both localization and classification, FEDN challenges the historical isolation of these tasks. Furthermore, the identification of discrepancies between expert and self-reported labels underscores the need for more nuanced annotation protocols in future datasets, suggesting that current benchmarks may not fully capture the complexity of human emotional expression. The work establishes a new standard for evaluating facial expression analysis as a holistic detection problem rather than a sequence of disconnected tasks.