💬 NLP

Bilingual Text-to-Motion Generation: A New Benchmark and Baselines

This paper introduces BiHumanML3D, the first bilingual text-to-motion benchmark, and proposes Bilingual Motion Diffusion (BiMD) with a Cross-Lingual Alignment strategy to significantly improve cross-lingual motion generation performance and enable zero-shot code-switching.

Original authors: Wanjiang Weng, Xiaofeng Tan, Xiangbo Shu, Guo-Sen Xie, Pan Zhou, Hongsong Wang

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wanjiang Weng, Xiaofeng Tan, Xiangbo Shu, Guo-Sen Xie, Pan Zhou, Hongsong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical robot dancer. Right now, this robot is incredibly talented, but it has a major limitation: it only understands instructions written in English. If you tell it, "Do a backflip," it spins perfectly. But if you tell it, "Fais une salto" (French) or "做后空翻" (Chinese), it just stands there confused, or worse, it does the wrong move entirely.

This paper is about teaching that robot to speak two languages (English and Chinese) fluently, so it can dance to your instructions no matter which language you use.

Here is the breakdown of their solution, using some everyday analogies:

1. The Problem: The "Lost in Translation" Dance

Currently, if you want the robot to dance to a Chinese command, you have to use a translator app to turn the Chinese into English first, and then feed it to the robot.

The Issue: Translation apps are great at words, but terrible at movement.
The Analogy: Imagine you tell a translator, "The person is doing a Tanglang Quan (a specific martial arts stance)." The translator might say, "The person is doing a fist." The robot hears "fist" and just punches the air, missing the specific leg stance and cultural nuance.
The "Code-Switching" Nightmare: Real people often mix languages in one sentence (e.g., "Walk forward and then do a jump"). Current robots get completely lost when languages are mixed up in a single sentence.

2. The Solution: Building a New Dance Floor (The Dataset)

To fix this, the researchers couldn't just use a translator. They needed a massive library of dance moves paired with descriptions in both English and Chinese.

The Analogy: Think of the old library (HumanML3D) as a bookshelf full of English dance manuals. They wanted a bilingual library.
How they did it: They used a team of "Super-Editors" (AI Large Language Models) to translate the English manuals into Chinese. But AI makes mistakes, so they added a layer of Human Editors to check the work.
The Result: They created BiHumanML3D, the first-ever "Bilingual Dance Dictionary." It contains thousands of motion clips, each with a perfect English description and a perfect Chinese description.

3. The Brain Upgrade: The "Universal Translator" (CLA)

Now that they had the data, they needed a new brain for the robot. They built a model called BiMD (Bilingual Motion Diffusion).

The Old Way: Imagine two separate brains. One brain only knows English dance moves; the other only knows Chinese. They don't talk to each other.
The New Way (CLA): They introduced a feature called Cross-Lingual Alignment (CLA).
The Analogy: Think of CLA as a universal remote control.
- Before, the "English button" and the "Chinese button" were on different remotes and didn't work the same way.
- With CLA, the researchers taught the robot that the feeling of the word "jump" in English and the feeling of the word "jump" in Chinese are actually the same button on the same remote.
- They forced the robot to learn that "He walks in a circle" and "他在逆时针转圈" (He walks in a counterclockwise circle) point to the exact same physical movement in the robot's mind.

4. The Magic Trick: Zero-Shot & Code-Switching

Because the robot learned this "Universal Remote" connection, it can do some amazing things:

Zero-Shot: If you train the robot only on Chinese data, it can still understand English instructions perfectly! It's like learning to drive a car in the US and then being able to drive in the UK without ever seeing a UK road sign, because you understand the concept of driving, not just the signs.
Code-Switching: You can say, "Walk forward, then do a TiaoYue (jump)." The robot understands the mix perfectly because it knows the "Universal Remote" connects the English "Walk" and the Chinese "TiaoYue" to the right muscles.

5. The Results: A Better Dancer

The researchers tested their new robot against the old ones.

Old Robots: When given mixed languages or Chinese, they stumbled, fell, or did the wrong moves.
The New Robot (BiMD): It danced perfectly. It was more accurate, moved more naturally, and handled language mixing like a pro.

In a Nutshell

This paper is about stopping the "language barrier" in computer animation. Instead of forcing a computer to translate your words before it understands your dance, they taught the computer to understand the meaning of the dance directly, regardless of whether you speak English, Chinese, or a mix of both. It's like giving the robot a soul that understands movement, not just a dictionary that understands words.

1. Problem Statement

Current text-to-motion generation systems are predominantly limited to English-only inputs. This creates significant barriers for global applications in gaming, filmmaking, and robotics, particularly for non-English speakers and culturally specific actions. The paper identifies three critical challenges in advancing bilingual text-to-motion generation:

Data Scarcity: There is a lack of high-quality, publicly available bilingual motion datasets. Existing datasets (e.g., HumanML3D) are English-only.
Inadequacy of Translation Pipelines: A "translate-then-generate" approach fails because standard translation tools often lose subtle motion semantics (e.g., specific body part grounding or directional nuances) and cannot handle code-switching (mixing languages within a single sentence), which is common in natural bilingual communication.
Encoder Limitations: General-purpose multilingual text encoders (e.g., mBERT, T5) are pre-trained on imbalanced corpora and lack motion-specific semantics, leading to performance gaps between languages and poor zero-shot cross-lingual generalization.

2. Methodology

The authors propose a comprehensive solution involving a new dataset and a specialized model architecture.

A. Dataset: BiHumanML3D

The authors introduce BiHumanML3D, the first bilingual (English-Chinese) text-to-motion benchmark.

Construction Pipeline: They extended the existing HumanML3D dataset using a multi-stage, LLM-assisted annotation pipeline:
1. Initial Translation: Large Language Models (LLMs) translate English descriptions to Chinese.
2. Refinement: A second LLM corrects artifacts like gender bias, literal translations, and unnatural phrasing.
3. Human Verification: Human annotators manually verify and correct entries, particularly those with high semantic complexity (e.g., multiple verbs), ensuring linguistic and contextual fidelity.
Scale: The final dataset contains 13,312 motion sequences with high-quality bilingual annotations.

B. Model: Bilingual Motion Diffusion (BiMD)

The proposed model is a unified diffusion framework designed to generate motion from either English or Chinese inputs (or mixed code-switched inputs) using a single model.

Cross-Lingual Alignment (CLA): This is the core innovation. Instead of using off-the-shelf multilingual encoders, the authors employ a Teacher-Student knowledge distillation approach:
- Teacher: A frozen, high-quality English-vision alignment model (OpenCLIP).
- Student: A multilingual model (XLM-Base) fine-tuned to align its Chinese (and English) embeddings with the teacher's English space.
- Loss Function: A symmetric Kullback-Leibler Divergence ( $L_{CLA}$ ) is used to minimize the distance between the student's embeddings and the teacher's embeddings, creating a unified conditional space where English and Chinese representations are interchangeable.
Latent Diffusion: The model builds upon the Motion Latent Diffusion (MLD) framework. It compresses raw motion into a latent space via a VAE and conditions the denoiser on the cross-lingually aligned text embeddings.
Training Strategy: The model is trained with bilingual diffusion, where each motion is randomly paired with either its English or Chinese description. This forces the model to learn shared motion patterns while respecting language-specific nuances.

3. Key Contributions

BiHumanML3D Benchmark: The first dedicated bilingual text-to-motion dataset, constructed via a rigorous LLM-assisted and human-verified pipeline.
Bilingual Motion Diffusion (BiMD): A unified model that eliminates the need for separate monolingual models or translation pipelines.
Cross-Lingual Alignment (CLA): A novel module that explicitly aligns semantic representations across languages, enabling robust zero-shot generation and code-switching support.
Comprehensive Evaluation: Extensive experiments demonstrating that the proposed method outperforms monolingual baselines and translation-based approaches.

4. Experimental Results

The paper evaluates performance using standard metrics: R-Precision (retrieval accuracy), FID (Fréchet Inception Distance, measuring generation quality), and Diversity.

Monolingual Performance (HumanML3D):
- BiMD with CLA achieves an FID of 0.045 (vs. 0.169 without CLA) and R@3 of 82.8% (vs. 80.8% without CLA).
- It sets a new state-of-the-art (SOTA) on the standard HumanML3D dataset, outperforming previous models like MLD, MDM, and MoMask.
Bilingual Performance (BiHumanML3D):
- BiMD with CLA significantly outperforms retrained baselines (MDM, MLD, Mo.Diffuse) using standard multilingual encoders.
- Code-Switching: The model successfully handles mixed-language inputs (e.g., "He walks in a 逆时针 circle"), a task where existing methods fail completely.
- Zero-Shot Transfer: When trained only on Chinese, the model can generate accurate motions for unseen English prompts (and vice versa), demonstrating strong cross-lingual generalization.
Ablation Studies:
- CLA Necessity: Removing CLA leads to a significant performance gap between English and Chinese and poor code-switching handling.
- Translation Baselines: Direct translation of Chinese prompts to English followed by generation results in a drastic drop in R-Precision (from ~77% to ~70%) due to semantic distortion and loss of motion-critical details.
- User Study: In a blind study, 68.1% of users preferred motions generated by the CLA model over non-CLA versions for semantic consistency in paired prompts.

5. Significance

This work addresses a critical gap in the field of human motion synthesis by moving beyond English-centric limitations.

Cultural Accessibility: It enables motion generation for non-English speakers and captures culturally specific actions (e.g., martial arts stances) that are often lost in translation.
Technical Advancement: It proves that a unified bilingual model with explicit semantic alignment is superior to translation pipelines or separate monolingual models.
Future Impact: By releasing the dataset and code, the authors provide a foundation for future research in multilingual generative AI, virtual animation, and human-robot interaction across diverse linguistic cultures.