Dual-View Optical Flow for 4D Micro-Expression Recognition - A Multi-Stream Fusion Attention Approach

1. Problem Statement

Micro-expression recognition (MER) is a critical task in affective computing, aiming to detect involuntary, brief, and low-intensity facial movements that reveal true emotions. However, MER faces significant challenges:

• Data Complexity: The shift to 4D data (3D mesh sequences + time) offers rich geometric information but introduces high dimensionality and computational burdens. Directly processing 3D meshes or point clouds requires complex architectures (e.g., PointNet++, 3D CNNs) that are often inefficient for real-world deployment.
• Subtlety of Motion: Micro-expressions involve minute muscle movements that are easily overshadowed by static facial textures, identity features, or noise (lighting, pose).
• Temporal Dynamics: Existing methods often fail to capture the specific temporal phases of micro-expressions (onset, apex, offset) or treat motion components (horizontal, vertical, magnitude) uniformly without adaptive weighting.
• Class Imbalance: Micro-expression datasets are typically highly imbalanced, making standard classification metrics and loss functions suboptimal.

2. Methodology

The authors propose a Dual-View, Triple-Stream Attention Network (MicroAttNet) that converts complex 4D mesh data into efficient 2D optical flow representations.

A. Data Pre-processing & View Separation

• Dual-View Extraction: Although the input is 4D mesh data, the method utilizes the high-resolution facial texture images (.jpg) associated with the mesh. Each frame is split into Left and Right facial views.
• Spatial Consistency: A fixed cropping strategy is applied per sequence. The bounding box with the largest area across all frames is selected and applied uniformly to every frame to ensure consistent alignment, minimizing variability caused by minor head movements.

B. Apex Frame Detection (Unsupervised)

Since ground-truth apex annotations are unavailable in the challenge dataset, an automatic detection method is used:

• Motion Intensity Calculation: Dense optical flow (Farneback algorithm) is computed between the onset (neutral) frame and subsequent frames for both views.
• Peak Identification: The frame with the maximum sum of flow magnitudes ($I_f = \sum \sqrt{u^2 + v^2}$) is identified as the Apex Frame. This is done independently for left and right views to ensure phase alignment.

C. Phase-Aware Optical Flow Decomposition

The sequence is decomposed into two distinct temporal phases based on the detected apex:

1. Onset-to-Apex: Rising intensity phase.
2. Apex-to-Offset: Falling intensity phase.
   For each phase, dense optical flow is computed, extracting three complementary channels:

• Horizontal ($u$)
• Vertical ($v$)
• Magnitude ($m$)

D. Triple-Stream MicroAttNet Architecture

The core network processes the $u$, $v$, and $m$ channels through three parallel streams:

1. Triple-Stream Backbone: Each stream passes through two convolutional blocks (Conv + BN + ReLU + Pool + Dropout) to extract features independently.
2. Fusion Attention Module: After each convolutional block, a fusion attention mechanism aggregates global context via Global Average Pooling (GAP). A shared Multi-Layer Perceptron (MLP) generates adaptive weights ($\alpha_h, \alpha_v, \alpha_m$) to dynamically recalibrate the importance of each motion stream based on the specific expression instance.
3. Squeeze-and-Excitation (SE) Block: A dedicated SE block is applied to the Magnitude ($m$) stream to further enhance the representation of motion intensity by modeling inter-channel dependencies.
4. Classification: Features from all three streams are flattened, concatenated, and passed through a fully connected layer for multi-label classification.

E. Training Strategy

• Loss Function: Focal Loss is used to mitigate class imbalance by down-weighting easy examples and focusing on hard, underrepresented samples.
• Augmentation: A Phase-Aware Temporal Augmentation strategy is employed, training on both rising and falling phases to improve temporal robustness.
• Optimizer: Adam optimizer with early stopping.

3. Key Contributions

1. Dual-View Optical Flow Approach: Successfully projects high-dimensional 4D mesh data into synchronized 2D dual-view optical flow maps, balancing computational efficiency with the preservation of geometric motion cues.
2. Triple-Stream MicroAttNet: A novel architecture that separates motion into horizontal, vertical, and magnitude streams, utilizing a Fusion Attention mechanism to adaptively weight these modalities rather than using naive concatenation.
3. Phase-Aware Decomposition: Explicitly modeling the onset-apex and apex-offset phases allows the network to capture the bell-shaped temporal dynamics of micro-expressions.
4. State-of-the-Art Performance: The method achieved 1st place in the 4DMR IJCAI Workshop Challenge 2025, significantly outperforming the official baseline.

4. Experimental Results

• Dataset: Evaluated on the 4DME dataset (24 subjects, 5 emotion categories: Negative, Positive, Repression, Surprise, Others) used in the 4DMR 2025 challenge.
• Metric: Macro-UF1 (Unweighted F1-score).
• Performance:
  • Proposed Method (Red-Green-Blue Team): 0.536 UF1 (Official Submission).
  • Internal Best Configuration (Late Submission): 0.554 UF1.
  • Official Baseline: 0.355 UF1.
  • Improvement: The proposed method outperformed the baseline by over 50%.
• Ablation Studies:
  • Removing the Fusion Attention module dropped the score to 0.524.
  • Removing the SE Block dropped the score to 0.518.
  • Removing both dropped the score to 0.510.
  • This confirms that both attention mechanisms contribute approximately 3.6 points of UF1 gain.
• Threshold Tuning: The optimal decision threshold for multi-label classification was found to be 0.20.

5. Significance and Conclusion

This paper demonstrates that 4D micro-expression recognition does not necessarily require computationally expensive 3D deep learning architectures. By intelligently projecting 4D data into dual-view optical flow and employing a multi-stream attention mechanism, the authors achieved a robust, interpretable, and highly efficient solution.

The results validate that:

1. Motion decoupling (separating $u, v, m$) is superior to treating motion as a single tensor.
2. Adaptive fusion (via attention) is critical for handling the nuanced differences in how micro-expressions manifest across different motion directions.
3. Phase-aware modeling significantly enhances the detection of subtle, transient facial dynamics.

The work sets a new benchmark for 4D MER, proving that combining multi-view cues with phase-specific motion modeling is a highly effective strategy for high-dimensional affective computing tasks.