🤖 machine learning

Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation

This paper demonstrates that for fine-tuning Vision-Language-Action models on mobile manipulators with heterogeneous joint spaces, selecting checkpoints based on per-group error rather than total Mean Squared Error is critical for avoiding the masking of specific joint failures and ensuring superior real-world robot performance.

Original authors: Pau Montagut Bofi, Mario García Blasco, Tessa Pulli, Markus Vincze

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pau Montagut Bofi, Mario García Blasco, Tessa Pulli, Markus Vincze

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a robot to do a complex job, like picking up a mug and moving it to a table. This robot has many different "limbs": a rolling base (wheels), a neck (head), a gripper (hand), and an arm. Each part moves differently and has a different job.

The paper by Montagut Bofi and colleagues from TU Wien tackles a tricky problem: How do you know when your robot is "good enough" to go out into the real world?

The Trap of the "Average" Score

Usually, when engineers train a robot, they look at a single number called the Total MSE (Mean Squared Error). Think of this like a student's overall GPA.

If a student gets an "A" in Math and an "A" in History, but a "D" in Gym, their GPA might still look great. But if that student is applying to be a professional athlete, that "D" in Gym is a dealbreaker, even if the GPA is high.

The authors found that in robotics, this "GPA trap" is real. A robot model might have a perfect "Total Score" because it learned to move its wheels and head perfectly. However, it might still be terrible at moving its arm. Because the "easy" parts (wheels) are so good, they hide the fact that the "hard" part (the arm) is failing.

The Experiment: Two Robots, One Lesson

The team tested two different robot brains (models) on a Toyota HSR robot, which has 11 different moving parts.

The "Small" Robot (SmolVLA): This was a smaller model trained specifically on data from this one robot.
- The Problem: As they trained it, the robot got really good at moving its wheels and head quickly. But the arm and gripper were slow to learn.
- The Result: The "Total Score" looked great, but the robot kept failing because it couldn't coordinate its arm properly. The wheels were "masking" the arm's mistakes.
The "Big" Robot (π0.5): This was a much larger, pre-trained model that had seen many different robots before.
- The Twist: The team tried to "fine-tune" this big model by only teaching it the final step (the action head) while freezing the rest of its brain.
- The Surprise: This fine-tuned version got a better Total Score than the original big model. By the math, it should have been the winner.
- The Reality: When they put it on the real robot, it performed worse than the original. Why? Because the fine-tuning process accidentally made the arm worse. The model got so focused on fixing the gripper and wheels that it forgot how to move the arm correctly.

The Solution: Check the "Subject Grades"

The authors' main discovery is simple but powerful: Stop looking at the Total Score. Start looking at the "Subject Grades."

Instead of one big number, they broke the error down by body part:

How well did the Arm do?
How well did the Gripper do?
How well did the Base (wheels) do?

They found that the "Subject Grades" were the only thing that predicted how the robot would actually perform on the real world.

For the small robot, the Base was the weak link.
For the big robot, the Arm was the weak link after fine-tuning.

The Takeaway

If you are building a robot with many different moving parts, don't just pick the model with the lowest overall error. You have to check which specific part is failing.

Analogy: Imagine hiring a chef. If you only look at their "Overall Cooking Score," you might hire someone who is amazing at chopping vegetables but terrible at seasoning the soup. You need to check the specific skills (chopping, seasoning, plating) to see if they are actually fit for the job.

In short: The paper argues that for complex robots, per-group error (checking each body part separately) is a much more reliable signal than the total error (the average) when deciding if a robot is ready to work.

Technical Summary: Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation

Problem Statement

Deploying pretrained Vision-Language-Action (VLA) models on mobile manipulators presents a unique challenge due to the heterogeneity of their action spaces. While most VLAs (e.g., OpenVLA, RT-2) are trained on fixed-arm robots with 6–7 Cartesian degrees of freedom, platforms like the Toyota Human Support Robot (HSR) require controlling 11 joints spanning an arm, gripper, head, and wheeled base.

The core problem identified is that standard evaluation metrics, specifically aggregate Mean Squared Error (MSE), obscure critical performance dynamics in these heterogeneous spaces. When 11 dimensions are collapsed into a single metric, joint groups that are easy to predict (e.g., gripper or head) can mask groups that are underfitting (e.g., mobile base or arm). Consequently, a checkpoint with the lowest total MSE may not correspond to the model that performs best on the physical robot, leading to counterintuitive deployment failures where a "better" offline metric yields a "worse" real-world outcome.

Methodology

Models and Setup

The study compares two VLA architectures fine-tuned for the 11-DoF HSR:

SmolVLA (450M parameters): A lightweight model based on SmolVLM2-500M, configured with a flow-matching action head for 11-dimensional output. It was pretrained on a subset of the LeRobot community data (109k episodes, fixed-arm only) and then fine-tuned on HSR-specific teleoperation data.
$\pi$ 0.5 (3.3B parameters): A larger model based on PaliGemma2 with a diffusion-based expert action head. It was pretrained on a diverse corpus including mobile manipulation across multiple embodiments.

Training Regimes

SmolVLA: Trained in two phases: (1) Pretraining on 109k HSR episodes for 20k steps, and (2) Task-specific fine-tuning on a pick-up task (3,971 episodes) for 50k steps.
$\pi$ 0.5: Evaluated using pretrained baselines (80k and 100k steps) and an expert-only fine-tuning regime where the 3B-parameter backbone was frozen, and only the ~300M parameter action head was trained for up to 5k steps. This approach was chosen due to memory constraints preventing full fine-tuning.

Evaluation Protocol

The authors propose a per-group error decomposition rather than relying on aggregate MSE. The 11-dimensional action vector is split into four functional groups:

Arm (5 joints)
Gripper (1 joint)
Head (2 joints)
Base (3 joints)

Offline evaluation involved computing Mean Squared Error (MSE) for each group separately on held-out data. Real-robot validation was conducted on the Toyota HSR with 60 trials (20 per model) on a pick-up task (mug and cracker box), graded on a 4-point rubric.

Key Results

1. SmolVLA: The Mobile Base Bottleneck

Convergence Dynamics: During fine-tuning, the gripper error decreased rapidly (92.9% improvement), while the mobile base error improved significantly slower (57.8%).
Performance Ceiling: The mobile base remained the dominant source of error, effectively setting the ceiling for the total MSE.
Real-World Outcome: The best SmolVLA checkpoint (40k steps) achieved a mean real-robot score of 3.5/4. Failures were characterized by accurate navigation but miscalibrated grasping (approaching the object but missing the grasp by ~20cm), consistent with the residual arm error being higher than the base error in some contexts, though the base was the primary training bottleneck.

2. $\pi$ 0.5: The Expert-Only Disconnect

Baseline vs. Fine-Tuned: The pretrained $\pi$ 0.5 baseline (80k steps) achieved the highest real-robot score of 4.0/4 (20/20 successes).
The Counterintuitive Result: Expert-only fine-tuning reduced the total MSE from 1.04 to 0.95 (a 9% improvement). However, this "better" model performed worse on the robot (3.75/4, 15/20 successes), with a statistically significant difference ( $p=0.010$ ).
Per-Group Analysis: The improvement in total MSE was driven by gains in the gripper and base groups. However, the arm group error degraded significantly (from 0.30 to 0.59 MSE). The real-world failures (failing to lift the object cleanly) correlated directly with this arm degradation, not the total MSE or base error.

3. Metric Reliability

Total MSE Failure: In the $\pi$ 0.5 experiment, the model with the lowest total MSE (Expert-only 3k) was the worst performing on the robot.
Per-Group Success: The offline metric that consistently predicted real-robot ranking was the arm-group error.
Statistical Significance: Mann-Whitney tests confirmed $\pi$ 0.5 80k significantly outperformed both fine-tuned variants.

Key Contributions

Identification of the "Aggregate MSE Trap": The paper demonstrates that for heterogeneous action spaces, minimizing total MSE can lead to the degradation of critical joint groups (like the arm) if their error magnitudes are smaller than other groups (like the gripper), causing the optimizer to sacrifice them.
Per-Group Error Decomposition: The authors introduce a systematic evaluation method that decomposes error by joint group (arm, gripper, head, base), revealing training dynamics invisible in aggregate metrics.
Real-Robot Validation of Offline Signals: Through 60 real-robot trials, the study empirically validates that per-group error is a more faithful signal for checkpoint selection than total MSE.
Bottleneck Identification: The work highlights that the "bottleneck" group varies by model and training regime:
- For SmolVLA, the bottleneck was the mobile base (slowest to converge due to lack of pretraining exposure).
- For $\pi$ 0.5 expert-only, the bottleneck was the arm (degraded due to optimization trade-offs in the fine-tuning process).

Significance and Claims

The paper concludes that per-group error is a more reliable signal than total MSE for selecting VLA checkpoints destined for deployment on robots with heterogeneous action spaces.

The authors argue that the disconnect between offline metrics and real-world performance is a predictable consequence of collapsing heterogeneous joint groups into a single metric. They assert that:

A checkpoint can appear competitive on aggregate metrics while failing on deployment-critical groups.
The specific joint group that limits performance depends on the model architecture and the training regime (e.g., pretraining corpus diversity vs. fine-tuning data scarcity).
For practitioners, reporting per-group error is essential to identify which group is the current bottleneck (whether it is the base in a new model or the arm in a fine-tuned expert) before trusting a model for deployment.

The study does not claim to solve the underlying optimization imbalance (e.g., via uncertainty weighting or gradient balancing) but identifies it as the likely driver of the observed dynamics, leaving the application of such techniques to future work. The comparison between SmolVLA and $\pi$ 0.5 is framed as a practical comparison of available checkpoints rather than a controlled architectural study, acknowledging differences in pretraining corpora.