SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling
The paper introduces SeMoCo, a semantic-first motion codec that separates motion tokens into semantic and kinematic components to improve both reconstruction accuracy and language-conditioned motion generation, alongside the creation of the large-scale -MotionVerse dataset.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Making a computer understand human movement is a problem that sits at the intersection of language and physics. For years, researchers have tried to teach machines to generate realistic motion from text descriptions, asking them to create a video of a person walking or jumping based on a simple sentence. To do this, computers need a way to translate the continuous, fluid flow of a body in space into a format they can process, much like turning a spoken sentence into a string of letters. Early attempts treated motion as a smooth, unbroken stream of data, but a more recent approach has shown that breaking movement down into discrete, individual units—like words in a sentence—allows computers to generate more complex and varied actions. However, a significant hurdle remained: the tools used to break motion into these units were designed primarily to rebuild the movement as accurately as possible, not to understand what the movement actually meant. They treated a subtle shift in weight and a major action like a squat as if they were the same kind of data, forcing the computer to learn the difference only through trial and error.
A team of researchers has introduced a new system called SeMoCo that changes how this translation happens. Instead of forcing the computer to guess the meaning of a movement while trying to rebuild it, this new method separates the two tasks from the very beginning. The system treats every moment of movement as a small packet of information containing two distinct parts: a primary code that captures the general meaning of the action, and a series of secondary codes that fill in the fine-grained details of how the body moves. This approach is inspired by how speech recognition works, where the core meaning of a word is separated from the specific nuances of a speaker's voice. By giving the computer a dedicated "slot" for the meaning of the action, the system can learn to predict what happens next based on the story of the movement, while a separate process handles the precise geometry of the joints.
The researchers built this system using a massive collection of human motion data they created, which they named Ω-MotionVerse. This dataset brings together nearly one thousand hours of recorded movement from various sources, including professional motion capture and video reconstructions, and standardizes them into a single, consistent format. They trained their new codec, SeMoCo, on this data to learn how to compress movement into these dual-layer packets. The result is a system that can reconstruct a person's movement with higher accuracy than previous methods, reducing the error in joint positions to a level that is barely noticeable. More importantly, when the researchers used these packets to generate new movements from text descriptions, the computer produced actions that were not only physically realistic but also semantically correct, matching the intent of the prompt with greater reliability.
The core innovation lies in how the system organizes information. In older methods, the computer had to figure out the meaning of a movement and the details of the joints at the same time, often leading to a compromise where one suffered for the sake of the other. SeMoCo avoids this by using a "semantic-first" approach. When the system looks at a short interval of movement, it first identifies the broad action, such as "sitting down" or "turning around," and assigns this to a specific token. It then uses a separate set of tokens to describe the exact trajectory of the limbs and the contact of the feet with the ground. This separation allows a language model to focus on the progression of the story—the sequence of actions—without getting bogged down by the complex math of every joint angle. The model predicts the next action based on the previous ones, and then fills in the specific details of that action, creating a generation process that is both efficient and precise.
To test their work, the researchers compared their system against several existing models using standard benchmarks for motion generation and prediction. In the task of simply rebuilding a recorded movement from its compressed form, SeMoCo outperformed all other methods, achieving the lowest error rates in measuring the distance between the predicted joints and the actual joints. When asked to generate new movements from text, the system showed strong results, particularly in its ability to match the text description with the generated motion. The researchers found that while larger models generally performed better at generating text-based motion, the advantage of size was not universal; for predicting future movement based on a short clip of past motion, a smaller, more specialized version of their model actually performed better. This suggests that the way the information is structured is just as important as the sheer size of the computer's memory.
The study also highlights a trade-off inherent in this new design. By prioritizing the semantic meaning of the movement, the system does incur a slight cost in the absolute precision of the reconstruction compared to a system that ignores meaning entirely. However, the researchers argue that this is a necessary and beneficial exchange for any application where the computer needs to understand and generate actions based on language. The system does not claim to have solved the problem of human motion generation completely, but it provides a clear path forward by showing that separating the "what" from the "how" leads to better results. The work demonstrates that when a computer is given a clear, structured way to understand the intent behind a movement, it can generate actions that are not only physically plausible but also genuinely responsive to the user's instructions.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.