HSC-VLA: Hierarchical Scene-Clearing for Robust Bimanual Manipulation in Dense Clutter

The paper introduces HSC-VLA, a hierarchical framework that enhances robust bimanual manipulation in dense clutter by decoupling high-level semantic reasoning from low-level execution through explicit scene-clearing, achieving significantly higher success rates than monolithic baselines in complex long-horizon tasks.

The Gist

Imagine you are trying to find a specific jar of peanut butter on a grocery store shelf that is absolutely packed with hundreds of other jars, boxes, and bottles. Some are falling over, some are shiny and reflecting light, and some are hiding behind others.

Now, imagine a robot trying to do the same thing.

Most current robots use a "super-brain" that tries to look at the entire messy shelf, read the instruction ("Get the peanut butter"), and figure out exactly how to move its arms all at once. The problem? The robot gets overwhelmed. It sees too much noise (the shiny boxes, the falling jars) and forgets what it's actually supposed to grab. It's like trying to solve a math problem while someone is screaming random numbers in your ear.

This paper introduces a new system called HSC-VLA (Hierarchical Scene-Clearing Vision-Language-Action). Think of it not as a single super-brain, but as a highly efficient team with two distinct roles: a Strategic Commander and a Specialized Mechanic.

Here is how it works, using simple analogies:

1. The Commander (The "Brain")

The high-level part of the system is like a smart project manager or a tour guide.

• What it does: It looks at the messy shelf and the instruction ("Get the peanut butter"). Instead of trying to move the robot's arms, it does something clever: it draws a mental "mask" or a highlighter over the things the robot doesn't need to see.
• The Analogy: Imagine the robot is wearing a pair of smart glasses. The Commander tells the glasses, "Ignore the shiny soda cans, ignore the falling cereal boxes, and ignore the red apples. Only show me the peanut butter and the empty space right next to it."
• The Result: The robot's view of the world suddenly becomes clean and simple. The clutter is effectively "erased" from its vision.

2. The Mechanic (The "Cerebellum")

The low-level part of the system is like a skilled surgeon or a precision mechanic.

• What it does: This part receives the "cleaned-up" view from the Commander. Because the distracting noise is gone, the Mechanic can focus 100% of its energy on the physics of the task: "How do I gently grab this jar without knocking over the one behind it?"
• The Analogy: If the Commander is the architect drawing the blueprint, the Mechanic is the construction worker laying the bricks. The worker doesn't need to worry about the traffic outside or the weather; they just focus on the clean blueprint they were given.
• The Tech: This part uses a "diffusion" model (think of it like a sculptor slowly chipping away stone to reveal a statue) to figure out the smoothest, safest hand movements.

3. The Teamwork Loop

The magic happens because these two talk to each other constantly.

• The Problem with Old Robots: If an old robot tries to grab something and fails, it often panics or gets confused because it's still looking at the whole messy shelf.
• The HSC-VLA Solution: If the Mechanic fails to grab the jar, it tells the Commander. The Commander then re-evaluates: "Oh, the jar moved behind a box. Let me update the mask to ignore the box and highlight the jar again." Then, it sends the new, updated "clean view" back to the Mechanic to try again.

Why is this a big deal?

The researchers tested this on real supermarket shelves packed with hundreds of items.

• The Competition: The best "single-brain" robots (monolithic models) succeeded only about 34% of the time in these messy conditions. They got lost in the visual noise.
• The Winner: HSC-VLA succeeded 86.7% of the time.

The Takeaway:
By separating the "thinking" (figuring out what to ignore) from the "doing" (moving the arms), the robot stops trying to be a genius at everything at once. Instead, it becomes a master of focus. It clears the mental clutter so it can perform the physical task with the precision of a human expert who knows exactly what to look at and what to ignore.

In short: Don't try to see everything. Just see what matters.

Technical Summary

Here is a detailed technical summary of the paper "HSC-VLA: Hierarchical Scene-Clearing for Robust Bimanual Manipulation in Dense Clutter."

1. Problem Statement

The paper addresses the critical failure of modern Vision-Language-Action (VLA) models when deployed in high-density, unstructured environments (e.g., supermarket shelves).

• The Core Bottleneck: Monolithic end-to-end VLA models suffer from a "representation bottleneck." When raw, high-dimensional pixels are encoded directly into latent action spaces, task-relevant signals become entangled with irrelevant background clutter.
• Consequences: This leads to an attention dilution effect, where the model allocates capacity to distractors rather than geometric structures required for manipulation. This results in catastrophic failures in long-horizon tasks, unstable grasping, and an inability to recover from physical resistance or occlusions.
• Specific Challenges: The environment features severe occlusions, irregular object arrangements, and optical artifacts (specular reflections), making it difficult for robots to distinguish task-critical objects from overwhelming visual noise.

2. Methodology: HSC-VLA Framework

The authors propose HSC-VLA, a hierarchical framework that decouples high-level semantic reasoning from low-level sensorimotor execution through an explicit Scene-Clearing Abstraction. The system mimics a biological separation between a "Brain" (planning) and a "Cerebellum" (execution).

A. High-Level Module: The "Brain" (Semantic Planner)

• Architecture: Utilizes a frozen, large-scale Vision-Language Model (VLM), specifically Qwen3-v1-235B-A22B-Instruct.
• Function:
  • Task Decomposition: Breaks down long-horizon natural language instructions into a sequence of executable subgoals ($P = \{g_1, g_2, \dots, g_N\}$).
  • Scene Clearing: Instead of identifying targets directly, the Brain identifies task-irrelevant regions (distractors) and generates 2D bounding boxes for them.
  • Mask Generation: These boxes are passed to a zero-shot segmentation model (e.g., SAM/Cutie) to create pixel-level masks ($Q_t$) that highlight distractors.
• Temporal Consistency: To avoid computational overhead and temporal inconsistency, the mask is propagated over time using a lightweight update module ($Q_t = K(I_t, Q_{t-1})$) rather than re-segmenting every frame.

B. Low-Level Module: The "Cerebellum" (Execution Policy)

• Architecture: A diffusion-based visuomotor policy trained via behavior cloning.
• Input: The policy receives filtered observations ($\hat{I}_t$) where the distractor masks are applied ($\hat{I}_t = I_t \odot (1 - Q_t)$), alongside proprioceptive state ($s_t$) and the active subgoal ($g_i$).
• Mechanism:
  • Perception-Action Consistency: The training data is processed with the exact same masking pipeline used during inference. This ensures the policy learns to operate in a "clutter-filtered perceptual subspace," making it invariant to environmental appearance shifts.
  • Action Generation: The policy outputs an action chunk (a sequence of future actions) to ensure temporal smoothness and stability.

C. Verification and Replanning

• A verification module checks subgoal completion after each action chunk.
• If a subgoal fails, the system does not terminate; instead, the Brain updates the plan (retries, revises spatial constraints, or replans) based on the failure, enabling robust recovery.

3. Key Contributions

1. Hierarchical Control Architecture: A novel framework that factorizes manipulation into symbolic reasoning (Brain) and sensorimotor execution (Cerebellum), allowing for long-horizon orchestration without sacrificing high-frequency responsiveness.
2. Mask-Based Scene Simplification: Introduction of a VLM-guided mechanism that systematically prunes task-irrelevant distractors, transforming raw RGB observations into geometry-focused representations.
3. Perception-Action Consistency Protocol: A principled alignment between offline training and online inference within a clutter-filtered subspace, demonstrating improved zero-shot robustness and failure recovery.

4. Experimental Results

The framework was evaluated on a real Inspire-Omni O1 bimanual robot in densely cluttered supermarket shelves and in the RoboTwin 2.0 simulation.

• High-Density Clutter Performance:
  • HSC-VLA achieved an aggregate success rate of 86.7% in high-density clutter.
  • This significantly outperformed the best monolithic baseline, $\pi_0$-Full FT, which achieved only 34.3% (a 52.4 percentage-point absolute improvement).
  • Other baselines like ACT and DP3 collapsed to near-zero success rates (<10%) under heavy clutter.
• Task-Specific Gains:
  • Grasping: 85% (vs. 75% for best baseline).
  • Placing: 78% (vs. 13% for best baseline).
  • Bimanual Manipulation: 97% (vs. 15% for best baseline).
• Long-Horizon Tasks:
  • Clutter Sorting: 72% success (vs. 40% baseline).
  • Restocking: 66% success (vs. 14% baseline).
• Ablation Studies:
  • Dynamic Clearing (online mask updates) was crucial, achieving 80% success in high-density tasks compared to 69% for static masks and 56% for no masks.
  • Static masks failed in long-horizon tasks (10% success) because they could not adapt to moving objects, whereas dynamic clearing maintained focus (72% success).

5. Significance and Impact

• Solving the "Clutter Problem": The paper demonstrates that explicit scene abstraction is superior to relying on the implicit attention mechanisms of monolithic transformers in complex physical environments.
• Robustness: By filtering out visual noise before the policy sees it, the system achieves a level of robustness in real-world retail scenarios that current state-of-the-art end-to-end models cannot match.
• Scalability: The approach suggests a viable path for deploying bimanual robots in logistics and service sectors where environments are dynamic and densely packed, moving beyond controlled lab settings to unstructured real-world applications.
• Limitations & Future Work: The system currently relies on the quality of the high-level segmentation; errors in mask initialization can propagate. Future work aims to improve temporal mask tracking and reduce the computational latency of the online masking pipeline.