Technical Summary: From Action Units to Emotions

Problem Statement

Facial Expression Recognition (FER) remains constrained by practical challenges such as illumination variations, pose differences, and the scarcity of annotated data. Existing deep learning models often suffer from poor cross-domain adaptability, overfitting, and insufficient interpretability. Furthermore, while Multimodal Large Language Models (MLLMs) offer advanced semantic understanding, they frequently lack the sensitivity to perceive subtle facial geometric deformations and micro-expressions. Traditional end-to-end fine-tuning of MLLMs on "video-label" signals often neglects the fine-grained intermediate representations (Action Units) that underlie emotional expressions, leading to suboptimal reasoning chains.

Methodology

The paper proposes a Two-Stage Fine-Tuning Framework with Decision-Level Fusion designed to bridge low-level facial perception and high-level emotional semantics.

1. Two-Stage Progressive Fine-Tuning

The core strategy decomposes FER into two sequential learning stages to achieve "triple decoupling": perception from reasoning, domain-agnostic knowledge from domain-specific knowledge, and physical observations from emotion labels.

• Stage 1: AU-Detection Fine-Tuning (Perception)

  • Data: Utilizes the CAS(ME)³ dataset, which provides frame-level Action Unit (AU) annotations.
  • Task: Reformulates AU recognition as a Visual Question Answering (VQA) task. The model is instructed to identify active AUs based on visual input, leveraging prior knowledge of the Facial Action Coding System (FACS).
  • Model: The Qwen3-VL-32B-Instruct multimodal large model is fine-tuned using LoRA (Low-Rank Adaptation) to update only a small subset of parameters (Query and Value matrices), preserving pre-trained visual-semantic knowledge while learning AU detection.
  • Objective: To establish robust, domain-agnostic perceptual capabilities for subtle facial micro-movements.

• Stage 2: Expression Recognition Fine-Tuning (Reasoning)

  • Data: Transfers to the FER-2013 dataset (static images with emotion labels but no AU annotations).
  • Task: The model, initialized with weights from Stage 1, is further fine-tuned to map detected AU combinations to emotion categories. The output format requires the model to reason from AUs to the final emotion label, maintaining the VQA structure.
  • Strategy: Only new LoRA adapters for emotion recognition are optimized; the visual encoder and Stage 1 adapters remain frozen to preserve the AU-to-emotion reasoning chain.

2. Decision-Level Fusion

To address the limitations of pure large models in perceiving fine-grained geometric deformations, the framework integrates the fine-tuned Qwen3-VL with OpenFace 3.0, a specialized lightweight CNN-based FER model.

• Mechanism: A weighted linear interpolation fuses the probability distributions ($P_{VLM}$ and $P_{OpenFace}$) of the two models:
  $$P_{final} = \alpha \cdot P_{OpenFace} + (1 - \alpha) \cdot P_{VLM}$$
• Rationale: OpenFace 3.0 provides sensitivity to local facial landmarks and geometric features, while the fine-tuned MLLM offers high-level semantic understanding and contextual reasoning.

Key Contributions

1. Validation of Large Model Baselines: Systematically evaluated the zero-shot and few-shot performance of Qwen3-VL-32B-Instruct on FER, establishing a strong baseline.
2. AU-Guided Two-Stage Fine-Tuning: Introduced a curriculum learning paradigm where the model first learns AU detection on CAS(ME)³ before mapping to emotions on FER-2013. This enables the model to acquire AU inference capabilities and reason through an interpretable chain ("AU inference first, emotion prediction second").
3. Heterogeneous Model Collaboration: Demonstrated the feasibility of decision-level fusion between a large multimodal model and a specialized small model (OpenFace 3.0), validating their complementary strengths.
4. Robust Empirical Performance: Showed that the proposed framework significantly outperforms non-fine-tuned baselines and standalone specialized models, particularly in improving recall for minority classes and enhancing interpretability.

Experimental Results

Experiments were conducted on the FER-2013 test set using metrics including Accuracy (ACC), Precision, Recall, and F1-Score.

• Performance vs. Baselines:
  • The Fine-tuned Qwen3-VL achieved 66.73% accuracy, an 8 percentage point improvement over the non-fine-tuned baseline (58.73%).
  • The Fused Model (Qwen3-VL + OpenFace 3.0) achieved 67.37% accuracy.
  • Compared to traditional CNNs, the fused model slightly outperforms GoogLeNet (67.04%) but falls behind ResNet-50 (69.33%), VGG-16 (68.75%), and DenseNet (69.38%).
  • Despite not surpassing the optimal specialized CNN models in overall accuracy, the fused model demonstrated superior Weighted Precision (69.47%), comparable to ResNet50 (69.84%).
• Advantages of Fusion:
  • The fused model showed a significant improvement in Macro-Recall (62.71%) compared to the pure large model (58.64%), indicating better capture of minority classes (e.g., disgust, fear).
  • Compared to OpenFace 3.0 alone (48.02% accuracy), the joint framework improved accuracy by 19.35 percentage points (40.3% relative improvement).
• Interpretability: The model successfully generated natural language explanations linking specific AU activations (e.g., AU4, AU7) to emotion judgments, providing a traceable reasoning chain absent in "black-box" CNNs.

Significance and Claims

The paper claims that the proposed two-stage fine-tuning strategy effectively elicits discriminative features for subtle facial geometric deformations and low-intensity expressions from large visual models, which typically struggle with such fine-grained details.

The authors assert that while pure large visual models possess formidable high-level semantic understanding, they exhibit inherent limitations in perceiving subtle facial deformations. The proposed method addresses this by:

1. Enhancing Interpretability: Using AUs as intermediate representations allows for traceable model outputs, increasing trustworthiness.
2. Improving Robustness: The decision-level fusion compensates for the lack of fine-grained spatial priors in large models, significantly boosting performance on minority classes and reducing bias toward head classes.
3. Providing a New Pathway: The study offers a technical paradigm for constructing highly robust and interpretable FER systems by leveraging the complementary strengths of MLLMs and specialized geometric models, validated by empirical results that show substantial gains over standalone inference of either model type.

The authors conclude that while the current accuracy does not yet surpass the absolute best specialized CNNs, the framework demonstrates clear superiority in prediction confidence, architectural interpretability, and the potential for synergistic large-model applications in affective computing.