iMiGUE-3K: A Large-Scale Benchmark for Micro-Gesture Analysis with Self-Supervised Learning

Technical Summary: iMiGUE-3K: A Large-Scale Benchmark for Micro-Gesture Analysis with Self-Supervised Learning

Problem Statement
Current affective computing and emotion understanding research predominantly relies on facial expressions and speech, often overlooking body language. While existing datasets exist for posed or spontaneous facial behaviors, there is a significant gap in large-scale resources for Micro-Gestures (MGs)—unintentional, subconscious movements driven by inner feelings (e.g., rubbing hands due to stress). These gestures are crucial for understanding suppressed emotions but are difficult to capture and analyze due to their spontaneous nature. Furthermore, existing gesture datasets are often small-scale, leading to overfitting and limited transferability in foundation models. Additionally, many existing datasets rely on sensitive biometric data, raising privacy concerns. There is currently no large-scale, in-the-wild dataset designed to support the pre-training of foundation models specifically for micro-gesture analysis.

Methodology
The paper proposes a comprehensive framework comprising a new dataset, a data sampling strategy, and a series of foundation models.

1. iMiGUE-3K Dataset Construction:

   • Source: Videos were collected from post-match press conferences of professional tennis players over seven years. This setting provides a natural emotional trigger (win/loss) and minimizes preparation time, eliciting spontaneous MGs.
   • Scale: The dataset contains 3,484 video clips (totaling >37 million frames) from 332 distinct subjects.
   • Annotation: It includes 32 micro-gesture classes organized into five body-part groups (Body, Head, Hand, Body-Hand, Head-Hand) and video-level emotion labels (Win/Loss).
   • Partitioning: The dataset is split into a labeled subset (341 videos, 17,800 manually annotated MG instances) for downstream evaluation and a large unlabeled subset (approx. 3,000 videos) for self-supervised pre-training.
   • Characteristics: The data exhibits a long-tailed distribution of gesture categories and covers diverse cultural and gender backgrounds.

2. Data Sampling Strategy for Self-Supervised Learning (SSL):

   • To utilize the unlabeled videos without exhaustive temporal annotations, the authors propose a sampling strategy to generate high-quality candidate clips.
   • Mechanism: The strategy samples clips based on an empirical duration prior derived from the labeled training set. It enforces facial-keypoint visibility constraints (ensuring anchors like the nose and eyes are detectable) and temporal non-overlap filtering to reduce redundancy and noise.
   • Output: This process yields 120K unlabeled skeleton clips suitable for pre-training.

3. MG-FMs (Foundation Models):

   • The authors introduce MG-FMs, a series of discriminative foundation models designed to learn transferable representations for MGs.
   • Architecture: The framework includes two modality-specific branches:
     • MG-FM-Skele: A skeleton-based model adapting the USDRL architecture with a dual-stream Dense Spatio-Temporal Encoder (DSTE). It utilizes feature decorrelation objectives (similarity, variance, cross-correlation) on augmented views (speed jitter, frame dropping, joint corruption).
     • MG-FM-RGB: An RGB-based model using a SMILE-style teacher-student masked video modeling framework. It reconstructs high-level semantic targets (extracted by a frozen ViCLIP encoder) rather than raw pixels, employing a two-stage progressive pre-training strategy.

4. Evaluation Protocols:
   The benchmark evaluates the models across five tasks:

   • Unsupervised MG recognition (Linear probing).
   • Semi-supervised MG recognition (Fine-tuning with 10% labeled data).
   • Supervised MG recognition (Full fine-tuning).
   • MG retrieval (Training-free nearest-neighbor search).
   • MG-based emotion recognition (Video-level binary classification: Win vs. Loss).

Key Contributions

1. iMiGUE-3K: The first large-scale, in-the-wild dataset specifically designed for micro-gesture analysis, supporting the pre-training of foundation models. It bridges the gap between behavior recognition and emotion understanding by linking MGs to emotional states (win/loss).
2. Broadened Benchmark: Extends the scope beyond closed-set classification to include MG retrieval and MG-based emotion recognition, moving toward instance-level and video-level affective analysis.
3. Novel Sampling Strategy: A scalable method to generate high-quality, MG-length candidate clips from unlabeled videos using empirical duration priors and visibility constraints, enabling effective SSL.
4. MG-FMs: A series of self-supervised pre-trained models (skeleton and RGB branches) that learn transferable representations for micro-gestures.
5. Comprehensive Evaluation: Extensive experiments demonstrating that large-scale self-supervised pre-training significantly improves performance across diverse downstream tasks, often matching or surpassing fully supervised baselines.

Results

• Unsupervised Recognition: The skeleton-based multi-stream variant (MG-FM-Skele) achieved 57.61% Top-1 accuracy via linear probing, performing on par with fully supervised 4-stream BlockGCN (57.85%). The RGB branch achieved 43.59%.
• Semi-Supervised Recognition: With only 10% labeled data, MG-FM-RGB improved significantly to 60.16% Top-1, while the skeleton branch remained competitive.
• Supervised Recognition: MG-FM-RGB achieved the highest accuracy among compared methods at 69.42% Top-1. A multimodal ensemble (combining skeleton and RGB) reached 71.60% Top-1, outperforming existing supervised ensemble methods.
• Retrieval: The skeleton model demonstrated superior training-free retrieval performance (40.06% Top-1 for motion stream) compared to the RGB model (22.56%).
• Emotion Recognition: In the video-level task (predicting win/loss), the proposed two-stream method achieved 0.64 accuracy, outperforming external baselines (0.63) that utilized more complex pipelines including facial features.

Significance and Claims
The paper claims that this work provides comprehensive tools for micro-gesture analysis and sets a solid foundation for future research in psychological diagnostics, affective computing, and human-computer interaction. By demonstrating that self-supervised learning on large-scale, privacy-preserving (non-biometric specific) data can yield representations highly transferable to downstream tasks, the authors argue that micro-gesture-based analysis is a viable and effective approach for improving emotion understanding. The work highlights that machines can accurately recognize and interpret these subtle, involuntary cues to infer emotional states, offering a privacy-preserving alternative to traditional biometric-based emotion recognition.