Technical Summary: MRBench

Problem Statement

Current human motion-text retrieval benchmarks, primarily HumanML3D and KIT-ML, suffer from critical limitations that hinder the reliable evaluation of cross-modal alignment. These datasets are dominated by homogeneous indoor motion capture (MoCap) scenarios, resulting in narrow and imbalanced motion distributions (e.g., "walking" constitutes a significant portion of samples). Furthermore, textual descriptions in these benchmarks are often oversimplified, repetitive, and lack semantic discriminability. A single text prompt frequently maps to hundreds of distinct motion instances, creating "text collisions" where standard retrieval metrics (like Recall@K) fail to distinguish between a model retrieving a valid motion versus one retrieving an equally valid but semantically redundant motion. Consequently, existing benchmarks cannot effectively assess cross-domain generalization or the sensitivity of models to varying levels of textual granularity (from concise summaries to fine-grained depictions).

Methodology

1. MRBench Construction

The authors introduce MRBench, a comprehensive benchmark constructed through a rigorous, multi-stage data curation pipeline based on the ViMoGen-228K corpus. The pipeline consists of four stages:

• Text-Guided Candidate Filtering: Rule-based filtering removes generic and repetitive expressions. Large Language Models (LLMs) score remaining captions for semantic discriminability and compactness, filtering out non-kinematic details (e.g., excessive scene or object descriptions) that distract from motion alignment.
• Taxonomy-Guided Balanced Sampling: Candidates are organized into a hierarchical taxonomy (7 coarse, 33 middle, 118 fine-grained categories). Stratified sampling ensures balanced coverage across categories, while semantic deduplication regulates uniformity.
• Semantic Alignment and Disambiguation: A Structured Semantic Alignment Evaluation (SSAE) using advanced vision-language models (e.g., Gemini-3-pro-preview) verifies that the primary motion described in the text is observable in the corresponding video frames. This step ensures unambiguous motion-text pairs. Underrepresented categories are supplemented using synthesized motions (HY-Motion) based on LLM-generated scripts.
• Multi-granular Text Expansion: Using visually grounded rewriting, each motion is annotated with three levels of text granularity: Concise (condensed semantics), Standard (balanced description), and Fine-grained (detailed body-part and temporal dynamics). These are manually validated to ensure consistency with the observed motion.

Dataset Statistics: MRBench contains 3,390 motion sequences covering 118 fine-grained categories. The data sources are heterogeneous: 43.8% MoCap, 26.5% in-the-wild video, 22.7% synthetic video, and 7.0% generated motions. This yields 10,170 captions in total.

2. Granularity-Aware Retrieval Model

To address the sensitivity of retrieval models to query granularity, the authors propose a lightweight, granularity-aware model:

• Anchored Base: A dual-encoder model (motion and text encoders with global projection heads) is first trained on standard-caption pairs and then frozen to serve as a global alignment anchor.
• Granularity-Specific Modules: For non-standard granularities (Concise and Fine-grained), lightweight modules are introduced:
  • Motion Extractors: Learnable query vectors perform attention pooling over motion tokens to extract granularity-specific representations.
  • Text Adapters: Granularity-specific projection adapters map text embeddings into the shared retrieval space.
• Pseudo-Supervision: The training set is expanded with LLM-rewritten concise and fine-grained captions (generated by Qwen2.5-7B-Instruct) to serve as pseudo-labels for training these specific modules.
• Inference Strategy: During inference, the text granularity dictates the active branch. For non-standard queries, a granularity-aware score fusion mechanism integrates the global similarity score with the adapted similarity score. This fusion maintains score comparability across different description levels without requiring paired standard captions at inference time.

Key Contributions

1. MRBench Benchmark: The first large-scale, multi-source benchmark specifically tailored for comprehensive human motion-text retrieval. It features heterogeneous motion distributions and natively supports cross-granularity evaluation (Concise, Standard, Fine-grained).
2. Data Curation Pipeline: A multi-stage pipeline combining rule-based filtering, taxonomy-guided balanced sampling, cross-modal semantic assessment, and multi-granular rewriting with human validation to ensure high-quality, unambiguous alignments.
3. Granularity-Aware Model: A retrieval architecture that freezes a standard-caption-aligned dual encoder while using LLM-rewritten captions to supervise granularity-specific extractors and adapters. This design improves performance on non-standard queries while preserving the performance of the standard-caption anchor.

Experimental Results

Extensive evaluations on MRBench reveal significant gaps in existing methods:

• Generalization Gap: Models trained on homogeneous datasets (like HumanML3D) exhibit substantial performance drops when evaluated on MRBench, indicating that in-domain success does not guarantee robustness to diverse motion sources.
• Granularity Sensitivity: Current baselines (TEMOS, TMR, MoPatch, SGAR) show pronounced sensitivity to query granularity, with performance degrading significantly for concise and fine-grained queries compared to standard ones.
• Model Performance: The proposed granularity-aware model, built upon the MoPatch anchor, successfully improves retrieval performance for fine-grained and mixed-granularity queries (Text-to-Motion and Motion-to-Text) without compromising standard-caption performance.
• Cross-Dataset Transfer: Models trained on the MRBench-Train corpus (40,168 pairs) achieve the strongest overall performance, suggesting that broader motion coverage and retrieval-oriented annotations are more beneficial for robust alignment than sheer data scale alone.

Significance

The paper posits that MRBench provides a necessary and comprehensive testbed for advancing motion-language alignment evaluation. By exposing the limitations of current benchmarks—specifically their inability to handle heterogeneous motion sources and multi-granular text—the authors argue that reliable progress in this field requires a shift toward diverse, balanced, and discriminative evaluation protocols. The proposed granularity-aware model demonstrates that robustness to non-standard descriptions can be achieved without sacrificing established alignment capabilities, offering a practical foundation for developing motion-language systems that generalize beyond narrow datasets and fixed caption styles.