Technical Summary: Vision-Language-Action Models on a Real-World UR5 Platform

Problem Statement

This project investigates the feasibility of transferring recent Vision-Language-Action (VLA) models from controlled research benchmarks to a real-world robotic platform, specifically a UR5e manipulator. While VLA models promise to unify perception, reasoning, and control into a single end-to-end framework, their practical deployment faces a significant gap between offline performance and unstable closed-loop behavior on physical systems. The study aims to determine whether open-source, fine-tunable VLA models (specifically OpenVLA and OpenVLA-OFT) can reliably execute manipulation tasks in real-world conditions, or if their failure stems from limitations in model architecture versus systemic issues in data representation, control interfaces, and pipeline integration.

Methodology

The research employs a structured, four-stage experimental workflow designed to isolate sources of failure, moving from controlled offline validation to real-world deployment:

1. Hardware and Software Setup:

   • Platform: A UR5e collaborative robot equipped with a Robotiq 2F-140 gripper.
   • Perception: A dual-camera system comprising a third-person "third-eye" camera (Intel RealSense D435) and a wrist-mounted "hand-eye" camera (Logitech webcam).
   • Control: An external client-server architecture where the client manages robot control via the ur_rtde library, and a remote server (cluster with NVIDIA A100 GPUs) handles model inference.

2. Data Acquisition and Engineering:

   • Acquisition Modes: Data was collected via two methods: scripted acquisition (waypoint-based trajectories with injected noise/variability) and freedrive acquisition (manual human guidance).
   • Dataset Versions: Five dataset versions (V0–V5) were created, progressively increasing in complexity (task diversity, viewpoint variation, noise injection) and shifting from scripted to human-guided trajectories.
   • Formatting: All data was converted to the Reinforcement Learning Datasets (RLDS) format, ensuring compatibility with OpenVLA training pipelines. Preprocessing included action computation (temporal differences), gripper discretization, and temporal synchronization.

3. Model Selection and Fine-Tuning:

   • OpenVLA: A 7B parameter model using a VLM architecture (DINOv2 + SigLIP encoders, LLaMA 2 backbone) with discrete action tokens (256 bins) and autoregressive decoding. Fine-tuned using LoRA.
   • OpenVLA-OFT: A refinement of OpenVLA featuring parallel action decoding, continuous action representation (via an MLP head), L1 regression loss, and multi-view inputs.
   • Baseline: A simpler supervised imitation-learning (behavior cloning) baseline using a ResNet18 backbone and MLP head was trained on the same datasets to isolate data/pipeline issues from model-specific limitations.

4. Evaluation Protocol:

   • Offline Sanity Check: Validating inference consistency using the Berkeley UR5 dataset.
   • Zero-Shot Deployment: Testing generalization on the real robot without fine-tuning (both in-distribution and out-of-distribution).
   • Fine-Tuning Evaluation: Assessing task-specific adaptation and closed-loop stability.
   • Robustness Tests: Injecting Gaussian noise and luminance shifts to test visual sensitivity.

Key Results

1. Offline vs. Closed-Loop Discrepancy

• Offline Performance: In controlled, offline settings, both OpenVLA and OpenVLA-OFT produced structurally coherent predictions that matched the general shape of ground-truth trajectories.
• Closed-Loop Failure: In real-world closed-loop execution, all models (including the imitation-learning baseline) exhibited progressive trajectory drift, stagnation, or instability.
• Zero-Shot Behavior: Without fine-tuning, the robot generated stable but invariant motions that failed to adapt to environmental changes or object positions, often stagnating after a short period.

2. Model-Specific Observations

• OpenVLA (Discrete): The model showed rapid training convergence but produced degenerate solutions insensitive to visual input. Robustness tests revealed that predictions relied on low-level statistical properties (e.g., global luminance) rather than task-relevant features. Even with post-processing alignment, the model systematically underestimated motion amplitude by ~30–40% due to action discretization.
• OpenVLA-OFT (Continuous): This model demonstrated more stable training dynamics and produced smoother, temporally coherent local predictions. However, it still suffered from long-horizon instability during recursive execution, indicating that continuous action representation alone does not resolve closed-loop drift.
• Imitation Learning Baseline: The simpler baseline produced more stable trajectories than the VLA models on the same data, suggesting that the instability is not solely a function of the VLA architecture but is inherent to the data-pipeline-control loop.

3. Data and Pipeline Sensitivity

• Preprocessing Inconsistencies: Mismatches between dataset construction preprocessing and inference pipelines significantly degraded performance.
• Action Representation: Ambiguities in coordinate frames and normalization/denormalization conventions were identified as critical failure points.
• Dataset Coverage: Datasets dominated by successful, scripted trajectories lacked recovery behaviors, limiting the model's ability to correct deviations during execution.

Key Contributions

The project establishes a reproducible framework for evaluating learning-based manipulation systems beyond simulation, providing four foundational assets:

1. A complete real-robot data acquisition pipeline for a UR5e platform.
2. A dataset conversion workflow aligned with RLDS standards.
3. An initial fine-tuning and inference infrastructure for VLA models (OpenVLA and OpenVLA-OFT).
4. A structured set of experimental observations grounded in real-robot trials.

Significance and Interpretation

The paper reframes the deployment of VLA-based robotics from a primarily model-centric challenge to a system-level problem. The central finding is that offline stability and model capacity do not guarantee closed-loop stability.

• System Integration over Model Capacity: The gap between promising offline indicators and unstable real-world behavior is attributed less to model limitations and more to the precise alignment of the entire data–model–control pipeline. Small inconsistencies in data representation, coordinate frames, or temporal alignment propagate and degrade performance.
• Limitations of Current Paradigms: The study highlights that current VLA formulations, even with improved action representations (continuous vs. discrete), lack mechanisms to enforce consistency under closed-loop dynamics. The compounding of small systematic biases leads to trajectory drift.
• Practical Implications: Successful deployment requires tighter integration between learning algorithms, control strategies, data acquisition protocols, and simulation-based iteration. The authors conclude that future progress depends on moving beyond one-step prediction metrics toward frameworks that explicitly account for closed-loop behavior and error accumulation.

The work emphasizes that while VLA models offer a compelling paradigm for flexible task specification, their practical effectiveness in real-world settings is currently constrained by system-level integration challenges rather than a lack of model sophistication.