💻 computer science

OMG: Omni-Modal Motion Generation for Generalist Humanoid Control

The paper presents OMG, a scalable, diffusion-based framework for generalist humanoid control that combines a hierarchical architecture with a meticulously curated dataset to enable state-of-the-art, omni-modal whole-body motion generation conditioned on language, audio, and reference motions.

Original authors: Siqiao Huang, Kun-Ying Lee, Dongming Qiao, Guanqi He, Zhenyu Wang, Yitang Li, Shaoting Zhu, Hang Zhao

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siqiao Huang, Kun-Ying Lee, Dongming Qiao, Guanqi He, Zhenyu Wang, Yitang Li, Shaoting Zhu, Hang Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A Brain and a Cerebellum for Robots

Imagine a humanoid robot (like the Unitree G1 used in this study) trying to learn how to dance, walk, or wave. Traditionally, engineers had to teach the robot one specific skill at a time, like teaching a dog to sit or stay. If you wanted the robot to do something new, you often had to start from scratch or write complex rules.

The authors of this paper propose a different approach, inspired by how the human body works. They split the robot's control system into two parts:

The Brain (OMG-DiT): This is the "planner." It takes high-level ideas (like "wave your hands" or music playing) and figures out what the robot should do next.
The Cerebellum (Motion Tracker): This is the "muscle memory." It takes the plan from the Brain and makes sure the robot's joints actually move smoothly and don't fall over.

The paper focuses on building a much smarter "Brain" that can understand many different types of instructions at once.

The Problem: Too Many Dialects, Not Enough Data

Before this paper, robot motion data was like a library with books written in different languages, some with missing pages, and some that were just scribbles.

Some data had text descriptions.
Some had music.
Some had video of humans dancing.
None of them were in a format the robot could actually use.

To fix this, the team created OMG-Data. Think of this as a massive translation and editing project. They gathered over 1,000 hours of motion data from various sources, cleaned it up, and translated everything into the "language" of their specific robot (the Unitree G1). They even used a physics simulator to check every single movement to make sure the robot wouldn't trip or break its own joints while trying to do it.

The Solution: The "Omni-Modal" Generator

The core of their system is called OMG-DiT. You can think of this as a universal translator and creative director rolled into one.

How it works:
Imagine you are at a party.

Text Input: Someone shouts, "Walk forward!" The Brain understands this and plans a walking path.
Audio Input: Someone plays a hip-hop beat. The Brain hears the rhythm and plans a dance that matches the beat.
Visual Input: Someone waves their arms. The Brain watches and plans to copy that wave.
Combo Input: Someone says "Dance to this beat!" while playing music. The Brain combines the meaning of the words with the rhythm of the music to create a unique dance.

The magic of OMG is that it doesn't need to be retrained for every new type of instruction. It uses a "shared backbone" (a central neural network) that has already learned the general rules of how a robot moves. When you give it a new type of instruction (like a new sensor or a new language), it just adds a small, lightweight "adapter" to connect that new input to the existing brain.

Key Achievements (What They Proved)

It's a "Foundation Model": Just like large language models (LLMs) can write essays, code, and poems because they learned from massive amounts of text, this model learned from massive amounts of motion data. Because of this, it can handle new tasks with very little extra training.
- Analogy: If you teach a human to ride a bike, they can figure out how to ride a skateboard much faster than someone who has never ridden anything. This robot does the same thing.
Zero-Shot Composition: The model can mix and match instructions it has never seen together before.
- Example: If it was trained to walk on text commands and dance to music separately, it can successfully follow a command to "Walk forward" while dancing to a specific song, even if it never saw that exact combination during training.
Real-World Execution: They didn't just simulate this on a computer. They put the system on a real Unitree G1 robot. The robot could listen to audio, read text, or watch a human, and immediately start moving in real-time without falling over.

The Results in Plain English

Better Quality: When asked to move based on text or music, their robot moved more naturally and accurately than previous methods.
Faster Adaptation: When they tried to teach the robot a new skill (like following a specific hand-tracking sensor called "Pico"), the pre-trained model learned it in minutes with very little data, whereas a model trained from scratch struggled and needed much more data.
Scalability: They found that making the "Brain" bigger (adding more computing power) made the robot move better, suggesting that this approach can keep getting smarter as we add more data and power.

Summary

The paper presents OMG, a system that gives humanoid robots a "generalist brain." Instead of hard-coding every move, the robot learns a universal language of movement from 1,000+ hours of data. This allows it to take instructions from text, audio, or human movement, mix them together, and instantly generate safe, physical movements that a real robot can execute. It's a step toward robots that can understand and react to the world as flexibly as humans do.

Technical Summary: OMG: Omni-Modal Motion Generation for Generalist Humanoid Control

Problem Statement

Humanoid whole-body control has advanced significantly through reinforcement learning, yet existing approaches remain constrained by two primary limitations:

Skill Specificity: Most policies are tied to specific tasks and reward designs, making them difficult to scale to general-purpose control.
Modality Rigidity: Motion tracking systems, while scalable, primarily replay given reference motions and lack the autonomy to interpret high-level, multi-modal human intent (e.g., language, audio, or new sensor inputs) without explicit reference data.

The authors argue that achieving general-purpose humanoid control requires a hierarchical structure mirroring biological motor systems: a scalable "brain" capable of reasoning with diverse conditioning modalities, situated atop a reactive "cerebellum" (motion tracker) that executes these plans. The core challenges in realizing this are the scarcity of high-quality, unified humanoid motion data and the difficulty of building a generator that supports compositional, extensible multi-modal inputs.

Methodology

The paper introduces OMG (Omni-Modal Motion Generation), a framework comprising a large-scale dataset and a unified diffusion-based model.

1. OMG-Data: A Unified Omni-Modal Corpus

To address data scarcity and heterogeneity, the authors curated OMG-Data, a corpus of over 1,174 hours of humanoid motion data unified in the Unitree G1 motion space. The pipeline involves:

Aggregation: Combining heterogeneous datasets from graphics and humanoid domains (e.g., AMASS, AIST++, BEAT2, MotionGV).
Retargeting: Using General Motion Retargeting (GMR) to map diverse human motion representations (SMPL, SMPL-X, FBX) to the Unitree G1 skeletal structure.
Annotation: Leveraging the Seed-1.8 vision-language model to generate structured text labels (action and style) for motion segments that lacked them.
Simulation-in-the-Loop Filtering: Executing retargeted motions in the MuJoCo simulator with a PD tracking controller. Trajectories are rejected if they result in falls (defined by root height and tilt heuristics) or violate joint limits, ensuring the dataset contains only physically feasible and safe trajectories.

2. OMG-DiT: Unified Diffusion Backbone

The core model, OMG-DiT, is a Diffusion Transformer designed to map multi-modal conditions into robot-executable future trajectories.

Architecture: It operates directly in the Unitree G1 motion space (125-dimensional feature vectors per frame) without a learned motion tokenizer. It uses an $x$ -prediction objective with a cosine noise schedule.
Conditioning Mechanism:
- Global Context: Language (via frozen T5 encoder) and motion history are injected as global context tokens via cross-attention.
- Frame-Aligned Signals: Audio and human reference motions are injected via FiLM (Feature-wise Linear Modulation) adapters, which modulate the features of specific frames.
- Extensibility: New modalities (e.g., Pico keypoints) can be incorporated via zero-initialized FiLM adapters, allowing the model to learn new interfaces without disrupting the pretrained motion prior.
Training Strategy: The model is trained with Classifier-Free Guidance (CFG) by randomly dropping conditioning modalities during training. This enables zero-shot composition of modalities at inference time (e.g., combining text and audio instructions).

3. Control Hierarchy

OMG functions as the high-level generator in a cascaded system:

Generator ( $\pi_\phi$ ): OMG-DiT predicts future whole-body reference trajectories based on multi-modal inputs and history.
Tracker ( $\pi_\psi$ ): A pre-trained motion tracker (HoloMotion) converts these references into executable joint-space actions on the robot.

Key Contributions

OMG Framework: A unified omni-modal motion generation framework that maps language, audio, human references, and their compositions into physically executable Unitree G1 trajectories.
OMG-Data: A large-scale, curated dataset of 1,000+ hours of omni-modal humanoid motion, aligned to a single robot embodiment through a rigorous retargeting and filtering pipeline.
OMG-DiT: A diffusion-based backbone supporting extensible conditioning. It demonstrates that new modalities can be added via lightweight adapters while reusing the shared motion prior.
Foundation Model Properties: The work establishes that humanoid motion generation exhibits foundation-model-like behaviors, including predictable scaling with model size, sample-efficient adaptation to new tasks, and zero-shot compositionality of control signals.

Experimental Results

The authors evaluated OMG across three primary modalities and several adaptation scenarios:

Text-to-Motion: OMG significantly outperformed baselines (GENMO, HYMotion, Kimodo) in motion quality (FID, Matching Score) and tracking fidelity (Fall rate, Joint-limit violations). For instance, OMG-XL achieved an FID of 6.03 compared to 24.68 for HYMotion.
Audio-to-Motion: OMG demonstrated superior audio alignment (BeatAlign) and motion quality compared to dedicated audio-to-motion models like LODGE and Bailando.
Human Reference-to-Motion: OMG substantially outperformed optimization-based (GMR, PHC) and learning-based (NMR, OmniRetarget) retargeting methods in both motion quality and tracking fidelity.
Sample-Efficient Adaptation:
- Few-Shot Text: When fine-tuning on unseen data (AMASS-CMU) with only 1% of the data, the pretrained OMG model achieved performance comparable to a model trained from scratch on 100% of the data.
- New Modalities: In Pico keypoint teleoperation, the pretrained model consistently outperformed scratch training across all data budgets (1%, 10%, 100%).
Scaling Behavior: Increasing model parameters from 50M to 500M resulted in consistent improvements in motion quality (MPJPE), confirming that motion generation benefits from increased model capacity.
Zero-Shot Compositionality: The system successfully executed complex instructions combining text and audio (e.g., "Raise your hands" while following a specific rhythm) without explicit training on such combinations.

Significance and Claims

The paper positions OMG as a concrete step toward foundation models for humanoid robots. The authors claim that generalist humanoid control can be advanced not merely by improving low-level controllers, but by scaling the motion-generation brain that interfaces human intent with physical execution.

Key claims include:

Generalizability: The framework successfully unifies diverse conditioning modalities under a single backbone, moving beyond task-specific policies.
Data Efficiency: Pretraining on a massive, unified corpus enables rapid adaptation to new tasks and modalities with minimal data.
Scalability: The system exhibits predictable scaling laws, suggesting that further improvements can be achieved through larger models and datasets.
Real-World Viability: The system is deployed in real-time on a Unitree G1, demonstrating that high-level generative planning can be effectively paired with reactive tracking for physical execution.

The authors acknowledge limitations, noting that current training data is primarily flat-ground motion and that the system currently uses a modular generator-tracker hierarchy rather than a fully joint-optimized end-to-end system. They suggest that jointly adapting the generator and tracker, or incorporating execution feedback, remains a direction for future work.