Technical Summary: RoboSemanticBench

Problem Statement

Vision-Language-Action (VLA) models are predicated on the assumption that the semantic competence of pretrained language or vision-language backbones should directly guide robot action prediction. Theoretical frameworks often describe these systems as dual-architectures: a low-frequency "System-2 Semantic Expert" that processes instructions and observations, and a high-frequency "System-1 Action Expert" that generates continuous control signals.

However, the paper argues that current VLA post-training pipelines may fail to preserve this semantic grounding. Robot fine-tuning datasets are significantly smaller and more task-specific than language pretraining corpora. Consequently, imitation learning losses often reward fitting conditional action distributions even when the semantic decision-making process is weakened or decoupled from the action generation pathway. Policies may learn "instruction-action shortcuts" or visual regularities rather than using language semantics to determine the correct physical action. While existing benchmarks measure task success, they often conflate motor execution, object recognition, and language grounding, failing to isolate whether the model truly understands the instruction's semantics to select the correct target.

Methodology: RoboSemanticBench (RSB)

To diagnose this specific failure mode, the authors introduce RoboSemanticBench (RSB), an embodied benchmark designed to test whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target.

Task Design

In each RSB episode, a robot is presented with:

1. A multiple-choice question (mathematical or general knowledge).
2. A set of candidate answer blocks physically visible in the scene.
3. An instruction to grasp the block corresponding to the correct answer.

The physical action is fixed as a single "pick-and-place" primitive. The semantic content varies across episodes, requiring the policy to:

• Solve the question (e.g., arithmetic or commonsense reasoning).
• Identify the correct option value.
• Bind that option to a specific visible block (which varies in position, color, and letter assignment per episode).
• Execute the grasp.

Crucially, the correct target is not tied to fixed visual features (like a specific color or position) but is determined dynamically by the question and the option-to-block mapping. This isolates whether instruction semantics participate in action prediction.

Benchmark Suites

RSB covers three semantic domains under both 4-choice and 10-choice configurations:

• RSB-Math: Procedurally generated arithmetic (addition, subtraction, multiplication) requiring computation rather than pattern matching.
• RSB-HardMath: Grade-school word problems derived from GSM8K, requiring extraction of quantities from natural language and compositional reasoning.
• RSB-General: Commonsense QA and MMLU-derived questions covering everyday tools, locations, and factual knowledge.

Metrics

To distinguish between low-level control failures and semantic grounding failures, the paper defines three metrics:

1. Task Success Rate (TSR): The fraction of episodes where the robot grasps the correct answer block.
2. Grasp Success Rate (GSR): The fraction of episodes where the robot grasps any candidate block.
3. Normalized Semantic Grounding (nSG): A score that factors out low-level grasping ability. It measures the probability of selecting the correct target conditioned on successfully grasping a candidate block, normalized against a random baseline ($1/N$).
   • $nSG = 0$ implies random selection among grasped blocks.
   • $nSG < 0$ implies selection worse than random.
   • $nSG > 0$ implies successful semantic grounding.

Key Contributions

1. RoboSemanticBench (RSB): A controlled diagnostic benchmark that transforms math and general-knowledge questions into embodied answer-selection tasks across six evaluation suites.
2. Diagnostic Metrics: The definition of GSR, TSR, and nSG to separate low-level grasping capabilities from semantic target selection, explicitly diagnosing whether instruction semantics participate in action prediction.
3. Empirical Evaluation: A comprehensive evaluation of representative VLA models (including OpenVLA, $\pi_0$, $\pi_0.5$, GR00T, etc.) revealing a persistent gap between backbone-level semantic competence and action prediction.
4. Intervention Analysis: Experiments testing whether CoT-style reasoning (ReasoningVLA) or VQA cotraining can close the semantic grounding gap.

Results

The evaluation of nine representative VLA models yields the following findings:

• Semantic Grounding Failure: Most models achieve high GSR (successfully grasping some block) but exhibit low TSR. When normalized for grasp success, the nSG scores for most models are near or below zero. This indicates that once a model successfully grasps a candidate, its choice of which candidate to grasp is effectively random or worse than random.
• The "Gap": There is a significant discrepancy between the models' ability to manipulate objects and their ability to use instruction semantics to select the correct object. This gap is particularly pronounced in the 10-choice suites.
• Exception ($\pi_0.5$): The $\pi_0.5$ model achieved the highest average TSR and the only clearly positive average nSG. The authors attribute this to its use of subtask annotations during pretraining, which may provide weak supervision for decomposing high-level instructions. However, even $\pi_0.5$'s nSG remains modest.
• Failed Interventions:
  • ReasoningVLA: Forcing the model to generate a Chain-of-Thought (CoT) before acting improved TSR slightly but did not reliably ground the semantic decision into the action pathway. In many failure cases, the CoT identified the correct answer, but the robot still grasped the wrong block.
  • VLA Cotraining: Adding VQA supervision during robot fine-tuning to preserve semantic competence actually reduced performance, suggesting a conflict between language-centric auxiliary supervision and the action-learning objective.
• Object Independence: Experiments replacing lettered blocks with everyday objects (toys, shoes) confirmed that the failure is not due to the simplicity of the "block" interface but is a fundamental issue of semantic grounding in action prediction.

Significance and Claims

The paper claims that RSB exposes a critical limitation in current VLA architectures: the disconnect between semantic understanding and action generation.

• Diagnosis over Evaluation: RSB is not merely a performance benchmark but a diagnostic tool. It reveals that high task success rates in other benchmarks may be ambiguous, potentially driven by visual shortcuts or dataset regularities rather than genuine instruction understanding.
• Training Limitations: The results suggest that simply attaching a strong pretrained VLM backbone to an action expert is insufficient. The current fine-tuning pipelines (imitation learning) often fail to route semantic decisions from the backbone into the action pathway.
• Future Direction: The paper posits that future VLA systems require training objectives and interfaces that explicitly preserve the selected semantic target and expose it to the action module in a stable, scene-grounded form. The goal is to achieve models where actions are genuinely grounded in instruction semantics, rather than relying on imitation shortcuts.

The authors conclude that while VLA models can learn the motor primitives for manipulation, they currently struggle to use complex instruction semantics to determine which action is correct, highlighting a persistent gap between the "Semantic Expert" and the "Action Expert."