💻 computer science

Fine-grained Human Motion Understanding with Language Models

This paper introduces \methodname, an LLM-based model that achieves state-of-the-art fine-grained human motion understanding by representing skeletal poses with explicit timestamps and leveraging diverse pose- and motion-level supervision, enabling superior performance on both 2D and 3D benchmarks without relying on ground-truth 3D motion capture.

Original authors: Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew, Jan van Gemert, Xucong Zhang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew, Jan van Gemert, Xucong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart robot how to understand human movement. Most robots today try to learn by watching videos, like a security camera recording. But videos are messy: they are huge files, they show clothes and backgrounds (which can be distracting or a privacy risk), and if you speed them up or slow them down, the robot gets confused about when things happened.

This paper introduces a new robot brain called FiGMo (Fine-Grained Motion understanding). Instead of watching a video, FiGMo looks at a "stick figure" skeleton of the person. But here is the magic trick: FiGMo doesn't just see the stick figure; it sees the stick figure with a clock attached to every single pose.

Here is how it works, broken down into simple concepts:

1. The "Timestamped Skeleton" Analogy

Imagine you are trying to describe a dance to a friend over the phone.

Old Way: You say, "He did a spin, then a jump, then a squat." Your friend knows the order, but they have no idea how long the spin lasted or if the jump was a quick hop or a slow leap.
FiGMo's Way: You say, "At 0.0 seconds, he starts spinning. At 2.4 seconds, he stops spinning and jumps. At 3.2 seconds, he lands in a squat."

FiGMo treats human motion exactly like this. It breaks movement down into a sequence of static poses, and it explicitly tells the AI, "This pose happened at this exact second." This allows the AI to answer questions like, "How long did the squat last?" or "What happened right before he stood up?" with incredible precision.

2. The "Universal Translator" (The Pose Encoder)

Usually, AI models are picky. Some only understand perfect 3D motion data captured in a lab with special suits (like a Hollywood motion-capture studio). Others only understand 2D drawings from regular cameras.

FiGMo has a special "Universal Translator" (called a Unified Pose Encoder). Think of it like a translator who speaks both "Lab-Suit" (3D) and "Phone-Camera" (2D).

If you feed it a messy, noisy 2D skeleton from a regular video, it cleans it up and understands it.
If you feed it perfect 3D data, it understands that too.
The Result: FiGMo is so good at this that even when it only uses 2D stick figures (which are less detailed), it beats other models that rely on expensive, high-quality 3D data.

3. The "School Curriculum" (Training Strategy)

To teach FiGMo to be so smart, the researchers didn't just dump a giant pile of data on it. They built a curriculum, like a school system:

Stage 1 (The Basics): They taught it to describe a single, frozen pose. "Look at this knee; it is bent."
Stage 2 (The Details): They taught it to answer specific questions about body parts. "Is the left arm higher than the right?"
Stage 3 (The Story): They started connecting poses into short sequences. "What action is happening here?"
Stage 4 (The Movie): Finally, they taught it to handle long, complex sequences with timing questions. "How long did the running phase last?"

The paper found that the most important part of this school was the variety of lessons. It was crucial to teach the robot about individual poses and full motion sequences. The "staged" approach (learning step-by-step) helped a little bit, but the main superpower came from the diversity of the training data.

4. Why This Matters (Privacy and Precision)

The paper highlights two main advantages:

Privacy: Because FiGMo only looks at the skeleton (the stick figure), it doesn't see the person's face, their clothes, or their background. It's like watching a shadow puppet show instead of a movie. This makes it safe to use in places where recording video is illegal or unethical, like a gym or a hospital.
Precision: Because it has the "clock" on every pose, it can catch subtle details. It can tell the difference between a slow, controlled squat and a fast, jerky one, which is hard for video-based AI to do.

The Bottom Line

FiGMo is a new way for computers to understand human movement. Instead of watching a blurry video, it reads a "skeleton script" where every movement is tagged with a timestamp. By teaching the AI to read this script using a mix of simple pose descriptions and complex motion stories, it has become the best at its job, even when using simple 2D data instead of expensive 3D data. It proves that you don't need a Hollywood movie to understand human motion; you just need the right "skeleton script" and a good clock.

Technical Summary: FiGMo – Fine-Grained Human Motion Understanding with Language Models

Problem Statement

Understanding human motion with fine-grained temporal precision is a critical step toward building human-centered AI systems for applications such as human-robot interaction, rehabilitation monitoring, and sports coaching. While recent advances in Large Language Models (LLMs) have expanded multimodal reasoning, existing approaches face significant limitations:

Video-Centric Limitations: Most methods rely on raw video inputs, which are computationally expensive, require aggressive temporal subsampling (obscuring rapid or subtle dynamics), and introduce privacy concerns due to identity and background information.
Loss of Temporal Grounding: Existing skeleton-based LLMs often compress motion into discrete tokens (e.g., via VQ-VAE) or concatenate pose sequences without explicit timestamps. This treats motion clips as atomic units, preventing the model from reasoning about the exact timing, duration, and rhythm of specific actions.
Data Constraints: Many prior works rely exclusively on clean, controlled 3D motion capture (MoCap) data, limiting their applicability to real-world scenarios where motion is derived from noisy 2D pose detectors on monocular videos.

Methodology

The authors propose FiGMo (Fine-Grained Motion), an LLM-based framework that explicitly models human motion as a temporally grounded sequence of skeletal poses.

1. Unified Pose Encoder

To handle both 2D and 3D inputs robustly, the authors adapt the motion encoder from MotionBERT.

Architecture: The encoder retains only spatial transformer blocks to encode individual static poses rather than continuous motion.
Pre-training: The encoder is pre-trained to reconstruct 3D poses from corrupted inputs (additive noise and random joint masking). It is trained on a mixture of detected 2D poses, 3D MoCap data, and 2D projections of 3D sequences. This yields a unified encoder capable of handling noisy 2D detections and clean 3D data consistently.

2. Multimodal LLM Structure with Temporal Grounding

The core innovation lies in how motion is represented to the LLM:

Pose Translator: A lightweight two-layer MLP projects pose embeddings into the LLM's semantic space.
Explicit Timestamping: Unlike naive concatenation of poses, FiGMo concatenates each pose embedding with a specific timestamp token (e.g., Time: 0.00s <pose>, Time: 0.03s <pose>). This allows the LLM to reason about motion order, duration, and rhythm at the pose level.
Optional Video Context: The architecture supports optional video input via a visual encoder to provide contextual cues without constraining the motion representation space.

3. Diverse Supervision Strategy

The authors investigate the necessity of different supervision levels by constructing a diverse training mixture spanning four modules:

Pose-Level Supervision: Includes pose captioning (PS-short, PS-fine) and pose question answering (PS-QA, PF-fine) derived from PoseScript and PoseFix datasets. These tasks focus on static body configurations and spatial relationships.
Motion-Level Supervision: Includes motion captioning (BABEL-cap) and motion question answering (HML3D-QA, BABEL-QA, FTM-QA). The FTM-QA module is specifically designed to supervise fine-grained temporal reasoning (duration, start/end times, ordering).
Training Schedule: The authors compare a single-stage training approach against a staged curriculum (Pose $\to$ Motion). The staged approach introduces pose-level tasks first, followed by motion-level tasks, to establish alignment before tackling complex temporal reasoning.

Key Contributions

FiGMo Architecture: An LLM-based method that explicitly represents motion as a sequence of timestamped skeletal poses, enabling fine-grained reasoning about action order, duration, and rhythm.
Unified 2D/3D Encoder: A pose encoder pre-trained on mixed and corrupted data, enabling the same model to operate on practical 2D detections and 3D MoCap sequences.
Supervision Analysis: An ablation study demonstrating that diverse pose- and motion-level supervision is the primary driver of performance gains, while staged training provides a smaller, additional benefit. Crucially, the study shows that pose-level supervision is essential for downstream motion understanding.
State-of-the-Art Performance: The method achieves top performance across multiple benchmarks, with a notable finding that a 2D-only variant outperforms previous 3D-based methods.

Experimental Results

FiGMo was evaluated on five benchmarks: BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D, and QEVD-Coach.

BABEL-QA (Motion QA): FiGMo (3D) achieved an overall accuracy of 0.758, outperforming previous state-of-the-art methods like HuMoCon (0.711). Remarkably, the FiGMo (2D) variant achieved 0.750, surpassing all prior 3D-based methods despite using only 2D skeletal input. Gains were particularly significant on temporally filtered queries ("Before", "After", "In Between").
HuMMan-QA: The method achieved the best performance across all metrics (Overall: 0.824 for 3D, 0.792 for 2D), demonstrating strong generalization beyond BABEL.
CompMo (Dense Captioning): FiGMo substantially outperformed UniMotion and DEMO in both captioning metrics (e.g., CIDEr: 456.95 vs. 134.44) and temporal localization (tIoU: 98.44 vs. 77.94), validating the effectiveness of explicit timestamp grounding.
NTU-RGB+D 120 (One-Shot Action Recognition): Using only 2D motion input, FiGMo achieved 69.2% accuracy, surpassing previous methods. Adding video as an auxiliary modality raised this to 77.4%.
QEVD-Coach (Exercise Feedback): The motion-based model outperformed video-only LLMs in providing feedback on exercise execution, highlighting the efficacy of explicit skeletal representations.

Significance and Claims

The paper claims that explicit temporal encoding and diverse supervision are the keys to fine-grained human motion understanding.

Privacy and Efficiency: By relying on skeletal poses rather than raw video, FiGMo offers a privacy-preserving and computationally efficient alternative suitable for edge devices and sensitive applications like rehabilitation.
Robustness to Input Modality: The ability of the 2D-only variant to outperform 3D-based baselines challenges the assumption that high-fidelity 3D MoCap is strictly necessary for fine-grained reasoning, provided the model is trained with appropriate supervision and robust encoding.
Supervision Hierarchy: The work establishes that pose-level understanding is a prerequisite for effective motion-level reasoning, suggesting that future motion-LLMs should incorporate static pose tasks even when the ultimate goal is temporal reasoning.

The authors conclude that while their method represents a significant step forward, future work could explore integrating scene and object context, as skeletal poses alone omit information about interactions with objects and 3D scene layout.