Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines

Technical Summary: Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines

1. Problem Statement

The rapid advancement of Vision-Language-Action (VLA) models—which map visual observations and natural language instructions directly to robot control actions—has shifted the research bottleneck from model architecture to data infrastructure. The authors identify three critical gaps in the current VLA landscape:

• Data Scarcity & Heterogeneity: Real-world data is high-fidelity but expensive and difficult to scale, while synthetic data is scalable but suffers from a "fidelity-cost trade-off" and sim-to-real gaps.
• Evaluation Deficiencies: Existing benchmarks often fail to disentangle whether a model's failure stems from poor low-level control, lack of long-horizon reasoning, or inability to handle compositional task complexity.
• Scaling vs. Grounding: While "data engines" (automated pipelines) can generate massive amounts of data, they often lack the physical grounding and embodiment alignment necessary to ensure that generated trajectories are executable in the real world.

2. Methodology (Taxonomy & Framework)

The paper proposes a data-centric taxonomy that organizes the VLA field into three interconnected pillars. This framework moves away from viewing data as a static resource and instead treats it as a dynamic ecosystem:

• Datasets: Categorized by source (Real-World vs. Synthetic) and analyzed through dimensions of embodiment diversity, modality composition (RGB, depth, tactile, audio), and action space formulation (End-Effector vs. Joint space; Absolute vs. Delta commands).
• Benchmarks: Analyzed via a two-dimensional landscape: Task Complexity (atomic vs. long-horizon/compositional) and Environment Structure (table-top vs. multi-scene/unconstrained).
• Data Engines: Classified by their generation paradigm:
  • Video-to-Data: Converting human videos into robot trajectories (e.g., via hand-to-robot retargeting or scene reconstruction).
  • Hardware-Assisted: Using teleoperation or wearable sensors (e.g., VR, gloves) to capture high-quality human-in-the-loop data.
  • Generative: Using LLMs, simulators, or diffusion models to synthesize new tasks, trajectories, or visual environments.

3. Key Contributions

• Unified Taxonomy: The first systematic survey to organize VLA research through a data-centric lens, providing a tree-structured taxonomy (2023–2025) to clarify the relationships between training, evaluation, and generation.
• Analytical Frameworks: Introduction of structured ways to characterize benchmarks (complexity vs. structure) and datasets (fidelity vs. cost).
• Identification of Structural Challenges: The authors distill four cross-cutting open problems:
  1. Representation Alignment: Bridging different robot embodiments.
  2. Multimodal Supervision: Integrating non-visual signals (tactile/force).
  3. Reasoning Assessment: Developing metrics for long-horizon, compositional logic.
  4. Scalable Data Generation: Ensuring physical realism in generative models.

4. Results & Analysis

The survey synthesizes findings from hundreds of recent works to reveal several industry trends and limitations:

• The Fidelity-Cost Trade-off: Real-world datasets (like Open X-Embodiment) offer breadth but suffer from interface heterogeneity, whereas single-platform datasets (like RT-1) offer consistency but lack scale.
• Reasoning Bottlenecks: Benchmarks like CALVIN and VLABench show that VLA models experience significant performance decay as instruction chains lengthen, indicating a failure in temporal abstraction and memory.
• The Scaling Imbalance: In data engines, the capacity to generate data (via LLMs or Diffusion) is currently outpacing the ability to verify the physical plausibility of that data, leading to "unreliable scaling."

5. Significance

This paper is significant because it advocates for a paradigm shift in Embodied AI: treating data infrastructure as a first-class research problem.

The authors argue that the path to General Purpose Robotics lies not in larger Transformers, but in the co-design of high-fidelity data engines and structured evaluation protocols. By highlighting the need to bridge the "sim-to-real quality gap" through high-fidelity scene reconstruction and predictive world models, the paper provides a roadmap for the community to move from narrow, task-specific robots to truly generalist embodied agents.