💻 computer science

GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs

This paper introduces GMoT, a Gated Motion-Aware Tokenization module combined with a progressive reward-guided policy refinement paradigm, to enhance Multimodal Large Language Models' ability to perform fine-grained micro-gesture video reasoning by explicitly distilling sparse kinematic evidence and anatomical grounding, achieving state-of-the-art performance on iMiGUE and SMG benchmarks.

Original authors: Taorui Wang, Wei Xia, Hui Ma, Zijia Song, Jiayu Zhang, Zeheng Wang, Yong Xu, Zitong Yu

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Taorui Wang, Wei Xia, Hui Ma, Zijia Song, Jiayu Zhang, Zeheng Wang, Yong Xu, Zitong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human body language. You show it a video of a person, and you want the robot to tell you exactly what that person is feeling or thinking based on a tiny movement. This is the world of computer vision, a branch of science where computers learn to "see" and interpret the visual world. Usually, these computers are great at spotting big, obvious actions like running, jumping, or waving. But there is a tricky corner of this world called micro-gestures. These are the super-subtle movements we all make without thinking: a quick twitch of a finger, a barely noticeable shrug of a shoulder, or a momentary tightening of the lips. They happen so fast and in such a small space that they are easy to miss.

The problem is that the smartest video-reading computers today, known as Multimodal Large Language Models (MLLMs), are like students who are very good at reading textbooks but terrible at watching a live play. They are so focused on the big picture—the static pose, the background, the person's face—that they often miss the tiny, fleeting clues that actually reveal the truth. They might look at a person with crossed arms and guess "angry," completely missing the fact that the person just tapped their finger once, which actually means "impatient." This paper tackles the challenge of teaching these powerful AI models to stop guessing based on the big picture and start paying attention to the tiny, moving details that matter.

The Problem: The "Static Snapshot" Trap

Think of a standard video AI like a photographer who takes a single, blurry snapshot of a movie and tries to guess the plot. If the movie is about a person shaking their shoulders nervously, a standard AI might just see a person standing still and guess "standing." It misses the motion because it looks at the video too slowly or averages everything out, washing away the tiny, fast movements.

The authors of this paper, Taorui Wang and his team, noticed that existing AI models were making "hasty guesses." They would look at a crossed-arm posture and immediately decide the person was "defensive," ignoring a subtle finger tap that might mean something entirely different. The models were relying on static appearances (what the person looks like) rather than kinematic evidence (how they moved). Even worse, because the models were only rewarded for getting the final answer right, they would sometimes "hallucinate"—making up a fancy-sounding reason for their guess that sounded logical but had nothing to do with what was actually happening in the video.

The Solution: GMoT (The "Motion Detective")

To fix this, the team built a new tool called GMoT (Gated Motion-Aware Tokenization). Imagine you have a very smart but slightly lazy detective (the AI model) who is trying to solve a mystery. The detective usually just glances at the crime scene and guesses. GMoT is like a specialized assistant who hands the detective a magnifying glass and a highlighter.

Here is how GMoT works, step-by-step:

Spotlighting the Action: Instead of looking at the whole video frame equally, GMoT acts like a spotlight. It scans the video to find exactly where the movement is happening (like a specific finger or a shoulder) and ignores the boring background noise.
Freezing the Motion: It takes two frames of the video that are right next to each other and subtracts one from the other. This is like taking two photos of a runner and subtracting the first from the second to leave only the "blur" of movement. This isolates the pure energy of the motion, stripping away the static parts of the person's body.
The "Gated" Injection: The team was worried that adding this new "motion information" might confuse the AI's existing brain. So, they built a "gate" (a control switch). At first, the gate is almost closed, letting the AI rely on what it already knows. As the AI learns, the gate slowly opens, letting in just the right amount of motion clues to help the AI make a better guess without overwhelming it.

The Training: Learning to Explain, Not Just Guess

The team didn't just want the AI to get the right answer; they wanted it to explain its reasoning based on what it actually saw. To do this, they created a special training process with four stages:

Warm-up: They taught the new GMoT module how to find motion on general videos first.
Domain Adaptation: They showed it thousands of micro-gesture videos so it learned the specific language of tiny movements.
Chain-of-Thought (CoT) Learning: They taught the AI to write out its thoughts before giving the answer. Instead of just saying "Shake Shoulders," it had to say, "I see the shoulders moving up and down quickly, which indicates..."
Reward-Guided Refinement: This was the final polish. They set up a scoring system that punished the AI for "lazy guessing" (picking the most common answer without looking) and rewarded it for focusing on the actual movement. If the AI tried to make up a story about the background or the person's mood without seeing the motion, it got a penalty. If it pointed to the specific body part moving, it got a reward.

The Results: Seeing the Invisible

The team tested their new system on two major datasets for micro-gestures: iMiGUE and SMG.

On the iMiGUE dataset, their model (based on the Qwen3-VL-8B backbone) achieved a 67.32% accuracy. This was a huge jump, improving the base model by 6.80 points.
On the SMG dataset, it reached 73.11% accuracy, a 3.11 point improvement over the base model.

These numbers mean the model is significantly better at spotting those fleeting, tiny movements that other models miss. But the team didn't stop at just the score. They introduced a new way to measure if the AI was actually "grounded" in reality. They checked if the AI's explanation mentioned the correct body part (like "lips" or "shoulders") when it got the answer right. They call this Body-Region Grounding (BRG) Recall. Their model showed it wasn't just guessing; it was actually looking at the right places.

Why This Matters

This paper suggests that for AI to truly understand human behavior, it needs to stop relying on static snapshots and start paying attention to the "flicker" of motion. By building a system that explicitly hunts for these tiny, transient clues and forces the AI to explain its reasoning based on them, the researchers have created a model that is less likely to hallucinate and more likely to understand the subtle, unspoken language of the human body. It's a step toward AI that doesn't just see what we look like, but understands what we are doing.

Technical Summary: GMoT for Fine-Grained Micro-Gesture Video Reasoning

Problem Statement

Micro-gesture recognition (MGR) involves detecting fleeting, spatially localized human movements (e.g., a slight shoulder shrug or a lip press) that reveal latent affective and cognitive states. These actions are characterized by low amplitude, short duration, and narrow spatial regions, often spanning only a few frames.

Current Multimodal Large Language Models (MLLMs), while powerful for general video understanding, struggle with MGR due to two primary limitations:

Static Bias: Standard video MLLMs rely on sparse temporal sampling and temporal pooling, which dilute or miss subtle kinematic signals. Consequently, models tend to rely on "static posture priors" (e.g., classifying a crossed-arm posture) rather than the discriminative motion cues.
Hallucinated Reasoning: Without explicit constraints, MLLMs often generate plausible-sounding but visually ungrounded reasoning traces ("hasty guessing") to satisfy the final classification label, failing to provide evidence-based explanations.

Existing evaluations focusing solely on Top-1 accuracy within a single dataset fail to verify whether models truly capture transferable micro-dynamics or merely memorize dataset-specific appearance biases.

Methodology: GMoT

The authors propose GMoT (Gated Motion-Aware Tokenization), a lightweight module designed to inject explicit kinematic bias into a vanilla MLLM without disrupting its pretrained visual-language interface. GMoT operates through three core mechanisms:

1. Motion-Aware Tokenization

GMoT distills sparse spatio-temporal cues into a compact motion token sequence before they are fused with the visual stream:

Motion-Prompted Spatial Saliency: A lightweight scorer projects patch tokens to generate relevance scores, applying a spatial softmax to create a weighted frame-level representation. This acts as a learned filter to amplify localized motion evidence while suppressing background noise.
Kinematics-Inspired Temporal Differencing: Instead of relying on complex temporal mixing, GMoT computes explicit adjacent-frame temporal differences. It decouples this into two streams: the mean magnitude (quantifying motion energy) and the signed difference (preserving directional kinematics). These are concatenated and projected into a fixed-length motion token.
Semantic-Preserving Adaptive Gating: To prevent disrupting the aligned visual-semantic space of the pretrained MLLM, the motion token is fused via a conservatively initialized semantic gate ( $\tilde{z}_t = z_t + \sigma(g) \odot m_t$ ). The gate starts in a "near-closed" regime, allowing the model to gradually learn how much motion evidence to introduce.

2. Training Strategy

The authors employ a four-stage progressive training pipeline to transition the model from simple classification to evidence-grounded reasoning:

Stage 0 (Warm-up): The GMoT module is trained on general video datasets while freezing the LLM and vision encoder to stabilize motion token generation.
Stage 1 (Video-Label Pairing): Supervised fine-tuning (SFT) on micro-gesture video-label pairs using LoRA to align visual representations with domain-specific semantics.
Stage 2 (CoT-format SFT): The model is trained to generate Chain-of-Thought (CoT) descriptions focused on observable motion evidence. This stage utilizes a semi-automatic annotation pipeline where multiple LMMs generate candidate descriptions, and a selector (Qwen3-VL-MoE) chooses the one that is class-correct, visually grounded, and free of hallucinations.
Stage 3 (Reward-Guided Policy Refinement): A reinforcement learning phase using Group Relative Policy Optimization (GRPO). The reward function ( $R_i$ $R_{i}$ ) combines four terms:
- Label Correctness ( $R_{label}$ ): Rewards exact matches, penalizes incorrect guesses, and severely penalizes out-of-domain hallucinations.
- Lazy-Prediction Penalty ( $R_{lazy}$ ): Discourages the model from exploiting long-tail distributions by repeatedly guessing high-frequency classes.
- Format Compliance ( $R_{format}$ ): Ensures the output follows the required CoT template.
- Observation Focus ( $R_{focus}$ ): Penalizes reasoning traces containing irrelevant background keywords.

Key Contributions

GMoT Module: A novel tokenization approach that integrates spatial saliency, explicit temporal differencing, and gated fusion into MLLMs, enabling them to capture subtle micro-gestures without heavy 3D convolutions.
Reasoning-Oriented Training: A framework combining semi-automatic CoT supervision with a progressive reward-guided policy refinement strategy. This encourages anatomical grounding and resistance to "lazy" appearance-based guessing.
Broader Evaluation Protocols:
- Body-Region Grounding (BRG) Recall: A new metric computed on correctly classified samples to quantify how often the model's reasoning correctly identifies the relevant anatomical region.
- Cross-Domain Transfer: An overlapping-label transfer protocol between the iMiGUE and SMG datasets to test robustness under dataset shift.

Experimental Results

The method was evaluated on the iMiGUE and SMG datasets using the Qwen3-VL-8B backbone.

Top-1 Accuracy: GMoT achieved 67.32% on iMiGUE and 73.11% on SMG. This represents an improvement of +6.80 and +3.11 points over the baseline Qwen3-VL-8B, respectively.
Comparison: The GMoT-augmented model outperformed all compared CNN, Transformer, and State-Space Model (SSM) baselines, as well as other MLLMs (e.g., InternVL2.5, Video-LLaMA2).
Robustness: The model maintained clear gains under label-preserving corruptions and demonstrated strong cross-domain transfer capabilities.
Grounding: The framework demonstrated high anatomical grounding in its generated rationales, as measured by the proposed BRG Recall.

Significance and Claims

The paper argues that the field of micro-gesture recognition requires a paradigm shift from purely label-driven recognition to evidence-based kinematic reasoning.

The authors claim that GMoT successfully addresses the "static bias" inherent in current video MLLMs by explicitly distilling weak local motion into compact tokens. By enforcing a training regime that penalizes hallucinations and rewards anatomical focus, the framework not only improves classification accuracy but also generates interpretable, visually grounded explanations. The work highlights that for fine-grained behavioral analysis, models must be forced to attend to what moved and how it changed, rather than relying on static appearance patterns.

The authors note that while the approach shows significant improvements, it is specifically designed for micro-gesture reasoning and does not inherently solve class-incremental learning challenges, though it draws inspiration from continual learning methods regarding representation adaptation. All code is set to be publicly released.