💻 computer science

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

This paper introduces Structured Motion Description (SMD), an encoder-free framework that converts joint position sequences into deterministic natural language text, enabling large language models to leverage their inherent world knowledge for state-of-the-art human motion understanding and reasoning without requiring cross-modal alignment modules.

Original authors: Yao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Speaking Different Languages

Imagine you have two friends who want to talk, but they speak completely different languages.

Friend A (The Computer Vision Model) speaks "Robot": It sees human movement as a stream of numbers—coordinates of joints, angles, and speeds. It's precise but cold and mathematical.
Friend B (The Large Language Model or LLM) speaks "Human": It is a super-smart AI that has read the entire internet. It understands stories, jokes, body parts, and directions perfectly. But it doesn't understand "Robot."

The Old Way (The Translator Problem):
Previously, to make them talk, researchers built a complex, expensive translator (called an encoder). This translator had to learn how to turn "Robot numbers" into "Human words."

The Flaw: This translator was like a clumsy interpreter. It often lost details, got confused if the data came from a different camera, and had to be retrained every time you changed the "Human" friend (the LLM). It was a heavy, complicated machine that needed a lot of training.

The New Idea: The "Biomechanical Translator" (SMD)

The authors of this paper said, "Why build a clumsy translator? Let's just speak the Human's language directly!"

They realized that doctors and physical therapists have been describing human movement in text for decades using biomechanics. Instead of saying "Joint 4 moved 15 degrees," a doctor says, "The left knee bent deeply, then straightened out."

The Solution: Structured Motion Description (SMD)
Instead of using a neural network to learn a translation, they wrote a simple rulebook (a computer script) that automatically converts raw movement data into a structured story.

Think of it like a smart recipe card that turns a chaotic cooking video into a clear list of steps:

The Ingredients (Raw Data): The computer takes the raw movement (e.g., a person kicking a ball).
The Rulebook (SMD): It calculates the angles and writes a structured report:
- Global Movement: "The person moved forward 1 meter."
- Body Parts: "The left hip raised from 3° to 81° (lifting the thigh)."
- Timing: "This happened between 0.0 and 0.9 seconds."
The Result: A clean, readable text document that looks like a medical report or a sports commentary.

How It Works in Practice

Now, you feed this text report directly into the LLM.

No Translator Needed: The LLM doesn't need to learn a new language. It just reads the text like a human would.
Lightweight Training: Because the LLM already knows what a "knee" is and what "lifting" means, you only need to give it a tiny bit of extra training (called LoRA) to teach it how to answer questions or write captions based on these reports. It's like giving a chef a new menu; you don't need to teach them how to chop vegetables again, just how to read the new menu.

Why This is a Game-Changer (The 3 Superpowers)

1. It's Smarter and More Accurate
Because the LLM is reading text it already understands, it uses its massive "world knowledge" to reason better.

Analogy: If you ask a robot to guess what a "kick" is based on numbers, it might guess "jump." If you ask an LLM to read a text saying "Leg extended rapidly," it instantly knows it's a kick.
Result: They beat all previous records in answering questions about motion and describing what people are doing.

2. It's "Plug-and-Play" (Flexible)
In the old days, if you wanted to switch from one LLM to a newer, smarter one, you had to rebuild the whole translator system.

With SMD: Since the input is just text, you can swap the LLM like changing batteries in a flashlight. You just take the same text report and plug it into a different LLM. You only need to train a tiny adapter (LoRA) for a few hours.
Analogy: It's like writing a letter in English. You can send that same letter to a friend in New York, London, or Tokyo. You don't need to rewrite the letter for each city; you just need a different stamp (the adapter).

3. It's Transparent (You Can See the Thinking)
Old methods were "black boxes." You couldn't tell why the AI thought a person was running.

With SMD: Because the input is text, you can look at the LLM's "attention map." You can literally see which words it focused on.
Analogy: If the LLM says "The person is waving," and you look at its attention, you see it highlighted the words "Right Shoulder" and "Elbow Flexion." You know exactly why it made that decision.

The Trade-off (The Catch)

There is one small downside: Speed.

Old Way: The translator compressed the movement into a tiny, dense code (like a ZIP file). It was fast to process.
New Way: The text report is long. It's like reading a full novel instead of a summary. This takes a bit more time for the computer to read (inference latency).
However: The authors argue that the massive gain in accuracy and the ease of using any LLM is worth the extra few seconds of reading time.

Summary

This paper is about stopping the translation and starting the conversation. Instead of forcing AI to learn a new, weird language of numbers, they simply describe human movement in plain English (structured text). This allows the super-smart LLMs to use their existing brainpower to understand motion better, faster, and more transparently than ever before.

1. Problem Statement

Current approaches to human motion understanding (specifically Motion Question Answering and Motion Captioning) rely heavily on encoder-based paradigms. These methods typically use a learned motion encoder (e.g., VQ-VAE, VAE, or linear projection) to map skeletal joint data into a latent space that aligns with a Large Language Model's (LLM) embedding space.

Key limitations of existing methods include:

Cross-modal Alignment Complexity: They require multi-stage training pipelines to align motion features with text, often coupling the system to a specific LLM backbone.
Lack of Interpretability: The learned motion tokens are opaque and not human-readable, making it difficult to analyze why a model makes specific predictions.
Fragility: Learned encoders are often sensitive to the data source (e.g., motion capture pipeline differences), leading to performance drops when applied to datasets different from their pretraining data.
Inefficiency: Adapting to a new LLM backbone requires retraining the entire encoder and alignment module.

2. Methodology: Structured Motion Description (SMD)

The authors propose Structured Motion Description (SMD), a deterministic, rule-based approach that converts raw joint position sequences directly into structured natural language text. This eliminates the need for learned motion encoders or alignment modules.

A. The SMD Conversion Pipeline (Stage 1)

The conversion function $f_{SMD}$ transforms joint positions ( $J$ ) into a text string ( $S$ ) through four deterministic steps:

Joint Angle Calculation: Following biomechanical conventions, the system computes 26 joint angles (e.g., hip flexion, knee flexion) based on bone vectors projected onto anatomical reference planes. This grounds the data in standard anatomical semantics (e.g., "hip flexion" = "raising thigh").
Global Trajectory Description: The pelvis trajectory is analyzed across translational (forward/backward, lateral, height) and rotational (yaw) axes. The time series is smoothed and segmented into intervals based on peak-valley detection, generating descriptions like "moves forward 0.00m → 1.23m [0.0s–2.5s]."
Joint Angles Description: Similar to trajectory, joint angle time series are segmented and labeled with temporal verbs: increases, decreases, holds, or repeats (for cyclic patterns).
Text Assembly: These components are assembled into a hierarchical text structure containing:
- Meta-information: Duration, frame count, FPS.
- Global Trajectory: Summary and per-axis segments.
- Joint Angles: Grouped by body part (e.g., [Left Hip], [Left Knee]).

B. LLM Fine-Tuning (Stage 2)

The resulting SMD text is fed directly into a pre-trained LLM as a prompt.

Training: Only LoRA (Low-Rank Adaptation) adapters are trained on the LLM's linear layers. The base LLM weights remain frozen.
Tasks: The same text input format supports both Motion QA (answering multiple-choice questions about body parts, directions, or actions) and Motion Captioning (generating a summary sentence).
No Encoder: No motion encoder, tokenizer, or projection layer is involved.

3. Key Contributions

Encoder-Free Paradigm: The first method to achieve state-of-the-art motion understanding by converting motion to text via deterministic rules, bypassing the need for learned cross-modal alignment.
LLM-Agnostic Flexibility: The same SMD text input works across different LLM architectures (tested on 8 models from 6 families, e.g., Qwen, LLaMA, Gemma, GLM) with only lightweight LoRA retraining (approx. 40M parameters).
Built-in Interpretability: Because the input is human-readable text, attention mechanisms can be visualized to show exactly which body parts or trajectory segments the model relies on for generation.
Robustness: The rule-based nature ensures consistent representation regardless of the motion data source (e.g., AMASS vs. HuMMan), unlike learned encoders which suffer from domain shift.

4. Experimental Results

The method was evaluated on BABEL-QA, HuMMan-QA (Motion QA), and HumanML3D (Motion Captioning).

Motion QA Performance:
- BABEL-QA: 66.7% accuracy (vs. 60.1% for previous SOTA IMoRe).
- HuMMan-QA: 90.1% accuracy (vs. 75.2% for IMoRe).
- Comparison: When compared to a controlled baseline (MotionGPT3-Qwen) using the same LLM backbone but a learned VAE encoder, SMD outperformed it by 16.6% on BABEL and 68.1% on HuMMan, proving the superiority of the text-based representation over learned latent spaces for cross-domain generalization.
Motion Captioning Performance:
- Achieved new SOTA on HumanML3D with R@1 of 0.584 and CIDEr of 53.16.
- Outperformed previous encoder-based methods (e.g., MotionGPT3, MG-MotionLLM) across all linguistic and alignment metrics.
Efficiency & Portability:
- Retraining on a new LLM backbone takes only 2–8 GPU hours (LoRA only), whereas prior methods require retraining full pipelines (hundreds of millions of parameters).
- Zero-shot performance (without fine-tuning) was already above random chance (35.6% on BABEL), demonstrating that LLMs inherently understand the biomechanical text.

5. Significance and Impact

Paradigm Shift: The paper challenges the prevailing "encoder-alignment" dogma in multimodal learning, demonstrating that for structured, low-dimensional data like human motion, deterministic text conversion is more effective than learned embeddings.
Accessibility: By removing the need for complex encoder training, the method lowers the barrier to entry for applying LLMs to motion data. Researchers can now plug motion data into any existing LLM with minimal adaptation.
Interpretability: The ability to trace model attention back to specific anatomical descriptions (e.g., "Left Hip Flexion") provides a level of transparency and debugging capability previously unavailable in black-box motion encoders.
Trade-off: The primary trade-off is inference latency due to longer token sequences (approx. 1,000–4,000 tokens vs. ~256 latent tokens), but the authors argue this is acceptable given the gains in accuracy, generalization, and interpretability.

In conclusion, SMD demonstrates that leveraging the LLM's pre-existing world knowledge of anatomy and physics through precise, rule-based textual descriptions is a superior strategy for human motion understanding compared to learning new cross-modal representations.