Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding

OBEYED-VLA improves the robustness of robotic manipulation in cluttered environments by disentangling perception from action through a framework that augments Vision-Language-Action models with task-conditioned, object-centric, and geometry-aware grounding.

The Gist

The Problem: The "Distracted Chef" Robot

Imagine you are teaching a robot to be a chef. You give it a simple instruction: "Pick up the ketchup bottle and put it in the bin."

Most current "smart" robots (called VLA models) are like a chef who has incredible muscle memory but a very easily distracted brain. If you place a jar of mustard, a bag of coffee, and a box of crackers right next to the ketchup, the robot gets confused. It sees "red things" or "bottles" and just grabs the first thing it sees. Even worse, if you tell it to "pick up the mustard" but there is no mustard on the table, the robot might just panic and grab the ketchup anyway because it’s "programmed" to always be grabbing something.

In technical terms, these robots suffer from "collapsed grounding." They are so focused on the action (moving the arm) that they stop paying attention to the meaning of your words.

---

The Solution: OBEYED-VLA (The "Laser-Focused Assistant")

The researchers created a new system called OBEYED-VLA. Instead of letting the robot look at the whole messy table at once, they added a "Perception Layer" that acts like a highly focused assistant standing next to the chef.

Think of it like this:

1. The "Highlighter" (Object-Centric Grounding)

Instead of the robot staring at a messy, cluttered countertop, the assistant takes a highlighter and marks only the things you mentioned. If you say "ketchup," the assistant highlights the ketchup and the bin, and then effectively turns off the lights on everything else (the mustard, the crackers, the messy background). The robot no longer sees a "cluttered table"; it only sees the "important objects."

2. The "3D Blueprint" (Geometry Grounding)

Even with the highlighter, a robot can get tricked by colors. If there is a bright red candy wrapper near the ketchup, the robot might grab that instead. To fix this, the assistant doesn't just show the robot a picture; they hand the robot a 3D blueprint (a depth map).

Imagine if, instead of looking at a photo of a bottle, you were feeling the shape of the bottle with your hands in the dark. You wouldn't care if the bottle was red, blue, or covered in a funny pattern; you would only care about its shape and size. This helps the robot recognize the object by its "skeleton" rather than its "outfit."

---

Why This is a Big Deal (The Results)

Because of this "Assistant," the OBEYED-VLA robot performs much better in the real world:

• The "Ignore the Noise" Test: When the table is covered in junk, the robot stays calm and only grabs the target.
• The "Honesty" Test: If you ask for something that isn't there (like mustard in a ketchup-only scene), the robot is smart enough to say, "Wait, there's no mustard here," and it does nothing.
• The "New Outfit" Test: If you show the robot a brand-new type of bottle it has never seen before, it doesn't get confused by the new color or label. It recognizes the shape and knows exactly what to do.

Summary in a Nutshell

Current robots try to see and act at the same time, which leads to chaos in messy rooms. OBEYED-VLA separates the two: it uses a smart "eye" to filter out the mess and focus on shapes, and then hands that clean, simple information to the "arm" to do the work. It’s the difference between trying to find a needle in a haystack while running, and having someone point a laser at the needle so you can just reach out and grab it.

Technical Summary

Technical Summary: OBEYED-VLA

Title: Clutter-Robust Vision–Language–Action Models through Object-Centric and Geometry Grounding
Authors: Khoa Vo, Taisei Hanyu, Yuki Ikebe, et al. (University of Arkansas, et al.)

---

1. Problem Statement

Current Vision-Language-Action (VLA) models (e.g., OpenVLA, Pi-0) typically employ an end-to-end monolithic paradigm where perception and control are optimized simultaneously for action prediction. The authors identify a critical flaw in this approach: the erosion of language-conditioned visual grounding.

Because these models are often fine-tuned on datasets with limited clutter or specific background contexts, they learn "shortcuts" (e.g., grasping any salient object regardless of the instruction). This leads to several failure modes in real-world cluttered environments:

• Distractor Sensitivity: The robot is easily distracted by irrelevant objects.
• Over-grasping: The policy executes actions even when the requested object is absent (absent-target failure).
• Background Overfitting: Performance degrades when the tabletop appearance or backdrop changes.
• Poor Generalization: Difficulty handling unseen objects or complex spatial relationships.

2. Methodology: The OBEYED-VLA Framework

To solve these issues, the authors propose OBEYED-VLA, a hierarchical framework that explicitly decouples perceptual grounding from action reasoning. Instead of feeding raw RGB images directly to the VLA, the framework transforms them into task-conditioned, object-centric, and geometry-aware observations.

The architecture consists of two primary stages:

A. Perception Grounding Module (Frozen)
This module acts as a "filter" to clean the visual input before it reaches the policy.

1. Object Segmentation Proposals: A fine-tuned YOLO11-Seg model identifies all potential objects and the robot arm in both the base and wrist camera views.
2. Object-Centric Grounding (Two-Stage):
   • Stage 1 (Base-View): An off-the-shelf VLM (Qwen3-VL) uses "set-of-mark" prompting (numeric markers on masks) to identify which objects are relevant to the natural language instruction. It then creates "canonical reference crops" of these objects.
   • Stage 2 (Cross-View Matching): To handle the perspective shift between the base and wrist cameras, the VLM uses the base-view crops as visual anchors to find the corresponding matching regions in the wrist-view.
3. Geometric Grounding: To prevent the model from relying on superficial textures or colors, the system uses Depth Anything v2 to estimate depth. It applies the task-relevant masks to these depth maps, providing the VLA with a 3D-structure-focused observation.

B. Action Reasoning Policy (Trainable)
The downstream VLA (e.g., Pi-0) is fine-tuned exclusively on clean, single-object demonstrations. Because the input it receives is already "cleaned" of clutter and background noise by the perception module, the policy learns to reason about geometry and task-relevant entities rather than raw pixels.

3. Key Contributions

• Decoupled Architecture: Proposes a method to strengthen VLA perception without requiring expensive, cluttered, or annotated synthetic datasets.
• Object-Centric & Geometry-Aware Inputs: Introduces a pipeline that uses VLM-driven cross-view reasoning and depth-based masking to isolate task-relevant information.
• Plug-and-Play Capability: The perception module is frozen and can be used with various existing VLA backbones.
• Efficiency Optimizations: Implements "object-centric grounding gating" (using ByteTrack to avoid redundant VLM calls) and parallel geometric processing to maintain real-time performance.

4. Experimental Results

The authors evaluated OBEYED-VLA on a UR10e robot across four challenging regimes:

• Distractor Objects: While standard VLAs collapsed to <10% success as distractors increased, OBEYED-VLA maintained ~80% success even with 7 distractors.
• Absent-Target Rejection: OBEYED-VLA achieved ~95% success in correctly not grasping when the requested object was missing, whereas baselines frequently performed spurious grasps.
• Background Shifts: The model remained highly stable (≥80% success) despite significant changes to tablecloths and backdrops, proving it does not overfit to environmental appearance.
• Unseen Objects: The framework demonstrated strong generalization, successfully manipulating novel object categories not seen during training.

Ablation Studies confirmed that:

• Two-stage grounding is vital for resolving wrist-view ambiguity.
• Geometric grounding (depth) is particularly critical for generalizing to unseen objects.

5. Significance

The significance of this work lies in its shift away from the "more data/larger model" brute-force approach to VLA training. By recognizing that perception and control have different optimization requirements, the authors provide a blueprint for making general-purpose robots more robust, reliable, and capable of operating in the messy, unpredictable environments of the real world. It demonstrates that explicit, modular grounding can bridge the gap between high-level semantic understanding and low-level motor control.