💻 computer science

LingoMotion: An Interpretable and Unambiguous Symbolic Representation for Human Motion

Inspired by the hierarchical structure of natural language, this paper proposes LingoMotion, an interpretable and unambiguous symbolic representation for human motion that defines a motion alphabet based on joint angles to construct words, phrases, and sentences for describing actions of varying complexity.

Original authors: Yao Zhang, Zhuchenyang Liu, Yu Xiao

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yao Zhang, Zhuchenyang Liu, Yu Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to dance. Currently, most robots learn by memorizing a giant, confusing list of numbers (like "move arm up 0.45, move leg down 0.12"). If you ask the robot to "dance faster," it has no idea what that means because it doesn't understand the concept of speed; it just sees a new list of numbers. It's like trying to explain a story by reading a string of random phone numbers.

The paper "LingoMotion" proposes a brilliant new way to teach robots: Give them a language.

Here is the simple breakdown of how it works, using a creative analogy:

1. The Problem: The "Black Box" vs. The "Language"

Old Way (The Black Box): Current systems are like a magic box. You put a video of a person walking in, and it spits out a secret code. You can't see inside the box to understand why the person is walking or how to change the speed. It's opaque and confusing.
The New Way (LingoMotion): This system treats human movement like spoken language. Just as we use letters to make words, and words to make sentences, LingoMotion breaks down movement into a structured language that humans and computers can both understand.

2. The "Motion Alphabet" (The Letters)

Think of a human body as a complex instrument. Instead of looking at where a hand is in the room (which changes if you walk across the room), LingoMotion looks at how the joints bend.

The Analogy: Imagine the alphabet. The letter "A" always sounds like "A," no matter who says it.
In Motion: A "Motion Letter" is a specific, standard bend of a joint. For example, a "Hip Flexion Letter" is simply the thigh moving forward.
Why it's better: If you walk forward, your hip bends the same way. If you walk backward, it bends differently. By measuring the angle of the bend (the letter), the system understands the movement regardless of where you are in the room.

3. Forming "Words" and "Phrases" (The Morphology)

Once you have the letters, you need to combine them to make meaning.

Motion Words: A single letter is just a bend. But when you combine a hip bend, a knee bend, and an ankle bend happening at the same time, you get a Word.
- Example: The "Walk" word is made of specific letters happening in sync.
Motion Phrases: This is where the magic of style happens. In English, you can say "Walk" or "Walk quickly." In LingoMotion, you take the "Walk" word and add attributes (like volume or speed).
- The Analogy: Think of a musical note. The note is the "letter." If you play it loudly and fast, it's a different "phrase" than if you play it softly and slowly. LingoMotion captures this by adding a "speed" and "size" tag to the letters.

4. Building "Sentences" (The Syntax)

Finally, you string these words together to tell a story.

Motion Sentences: A complex activity like "Playing Basketball" isn't just one move. It's a sentence: Dribble (Word 1) + Jump (Word 2) + Shoot (Word 3).
The Syntax: Just as grammar rules tell us that "The cat sat" makes sense but "Sat the cat the" does not, LingoMotion learns the rules of how movements flow together. It knows you usually jump before you shoot a basketball, not after.

5. Why This Matters (The Results)

The researchers tested this on a massive dataset of human movements (Motion-X).

The Test: They broke down a video of a person running into these "letters," threw away the video, and tried to rebuild the run using only the letters and their rules.
The Result: They rebuilt the run with 97% accuracy.
The Takeaway: This proves that you don't need a giant, messy video file to describe a movement. You can describe it perfectly using a simple, interpretable "sentence" of joint angles.

Summary

LingoMotion is like translating a chaotic, continuous movie of a person moving into a clear, readable book.

Letters = Joint bends.
Words = Simple actions (like walking).
Sentences = Complex activities (like playing sports).

This makes it possible for computers to truly understand movement, allowing us to edit, analyze, and generate human motion with the same ease we use to write a text message. Instead of a robot guessing what "run fast" means, we can simply tell it: "Use the 'Run' word, but set the speed attribute to 'High'."

1. Problem Statement

Current human motion representation methods face two critical limitations:

Lack of Interpretability: State-of-the-art models (e.g., MotionGPT, VideoGPT, ACTOR) rely on "black-box" latent vectors derived from joint positions. These encodings are opaque, making it difficult to diagnose, control, or understand specific motion features.
Ambiguity and Loss of Temporal Continuity: Existing approaches often use fixed temporal windows, which can disrupt the continuity of sub-movements. Furthermore, rule-based textual descriptions (e.g., PoseScript) often group distinct poses under identical labels, leading to ambiguity, and fail to capture the dynamic temporal nature of motion.
Global Dependency: Joint-position-based representations are sensitive to global translation and orientation, whereas biomechanical analysis suggests joint angles provide a more robust, intrinsic framework.

2. Methodology: LingoMotion Framework

The authors propose LingoMotion, a hierarchical "motion language" inspired by the structure of natural language (letters $\to$ words $\to$ phrases $\to$ sentences). This framework decomposes continuous joint-angle signals into discrete, interpretable symbolic units.

A. Motion Alphabet (The "Letters")

Basis: Instead of 3D joint positions, the system uses joint angles calculated based on biomechanical standards (using the SMPL model with 22 joints).
Rationale: Joint angles capture periodicity and are invariant to body shape and global orientation.
Discovery Pipeline:
1. Segmentation: Continuous joint angle signals are segmented at local extrema (transitions between positive and negative changes).
2. Clustering: Segments are normalized and clustered (e.g., using K-Means) to identify canonical shapes. Each cluster center represents a unique "motion letter."
3. Attribute Association: Each letter is associated with continuous attributes to ensure high-fidelity reconstruction:
  - Scale ( $\sigma$ ): Amplitude.
  - Bias ( $\mu$ ): Minimum value (ensuring continuity).
  - Length ( $l$ ): Duration.
- Result: A sequence of discrete symbols $(s, \sigma, \mu, l)$ representing the motion.

B. Motion Words, Phrases, and Sentences

Motion Words: Formed by coordinating relevant motion letters (e.g., simultaneous flexion of the left and right legs) to describe simple actions like walking.
Motion Phrases: Created by adjusting the attributes (scale, bias, length) of words to describe variations in speed or scale (e.g., "walk fast" vs. "walk slowly").
Motion Sentences: Complex activities (e.g., "playing basketball") are constructed by sequencing words and phrases. The temporal relationships (sequential or probabilistic co-occurrence) are modeled using advanced sequential models like Transformers.

3. Key Contributions

Novel Motion Language: Introduction of LingoMotion, a symbolic representation framework that mimics natural language structure to provide interpretable and unambiguous motion descriptions.
Segment-Cluster-Reconstruct Pipeline: A proposed methodology to discover a "motion alphabet" from raw data, transforming continuous signals into discrete, biomechanically meaningful units.
High-Fidelity Reconstruction: Demonstration that complex motions can be decomposed into unique sequences of symbolic units and reconstructed with high accuracy, bridging the gap between symbolic abstraction and raw data.

4. Experimental Results

The authors evaluated the framework using the Motion-X dataset (81.1K motion sequences).

Conversion Fidelity: The conversion from joint positions to joint angles (and back) was tested.
- Result: Joint positions were recovered with an $R^2$ of 0.978 and RMSE of 0.009, confirming the high fidelity of the angle-based representation.
Symbolic Reconstruction: The system encoded test data into discrete motion letters and reconstructed the joint-angle signals.
- Result: The reconstruction achieved an $R^2$ of 0.716 and RMSE of 0.921 (normalized). While lower than the direct position-angle conversion, this demonstrates that the discrete symbolic representation retains significant detail and is capable of reconstructing the original motion dynamics.
Alphabet Size: In the preliminary experiment, the alphabet consisted of 28 separate codebooks (one per joint angle channel), with each codebook containing between 12 and 20 unique "letters."

5. Significance and Future Work

Significance: LingoMotion shifts the paradigm from opaque latent spaces to interpretable, compositional, and unambiguous representations. This allows for precise motion analysis, editing, and comparison based on biomechanically meaningful components. It addresses the "black-box" problem in generative motion models.
Future Directions:
- Refining cluster centers to ensure they represent the most meaningful motion primitives.
- Investigating the convergence of the alphabet size as datasets expand.
- Developing models to handle variable-length sequences of motion words.
- Aligning the "motion language" with natural language to uncover syntactic patterns between motion and text (e.g., motion captioning).

In conclusion, LingoMotion offers a foundational framework for treating human motion as a language, enabling a level of control and interpretability previously unattainable with latent vector-based approaches.