🤖 AI

Visual Prompt Guided Unified Pushing Policy

This paper proposes a unified pushing policy that integrates a lightweight visual prompting mechanism with flow matching to generate reactive, multimodal actions, enabling efficient and versatile object rearrangement across diverse planning scenarios.

Original authors: Hieu Bui, Ziyan Gao, Yuya Hosoda, Joo-Ho Lee

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hieu Bui, Ziyan Gao, Yuya Hosoda, Joo-Ho Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a crowded dinner table, and your job is to tidy up. You have a pile of mixed-up plates, cups, and napkins. You could try to pick up every single item one by one with your hands, but that's slow and clumsy. Instead, you might use your elbow to gently nudge a stack of plates together so you can grab them all at once, or slide a stray napkin into a corner.

This paper is about teaching a robot to do exactly that: push things around to organize a scene, but with a superpower that makes it incredibly smart and flexible.

Here is the breakdown of their invention, explained simply:

1. The Problem: The "One-Tool" Robot

Most robots that push things are like a person who only knows how to use a hammer. If they need to push a block to a specific spot, they can do it. If they need to push two blocks together, they might fail. If they need to push one block away from a messy pile, they get confused.

The Old Way: Scientists built different robots for different jobs. One robot for "moving things," another for "grouping things," and another for "separating things." If the task changed, you had to swap the robot's brain.
The Limitation: Real life is messy. Sometimes you need to move, sometimes group, sometimes separate. Switching brains is slow and inefficient.

2. The Solution: The "Universal Pusher" with a Magic Wand

The authors created a Unified Pushing Policy. Think of this as a single, super-smart robot brain that can do all three pushing jobs (moving, grouping, separating) instantly.

But how does it know which job to do right now? They introduced a Visual Prompting Mechanism.

The Analogy: The Magic Wand and the GPS
Imagine the robot is a driver, and the human (or a high-level computer) is the passenger holding a Magic Wand.

The Wand (Visual Prompt): The passenger points the wand at a specific object on the table (like a red block) and then points to a destination (like a blue zone).
The GPS (Task Specifier): The passenger also whispers a command: "Move it," "Group it," or "Separate it."

The robot looks at where the wand is pointing and listens to the whisper.

If the whisper says "Move," the robot pushes the pointed object to the pointed spot.
If the whisper says "Group," the robot pushes the pointed object toward another pointed object to make a neat pile.
If the whisper says "Separate," the robot pushes the pointed object away from the messy pile so it's alone and easy to grab.

3. How It Learned: The "Flow" of Water

The robot didn't learn by reading a manual or doing math equations. It learned by watching humans.

The Training: Humans used a remote control to push blocks around while the robot watched. The robot recorded thousands of these moves.
The Secret Sauce (Flow Matching): Instead of trying to guess the next move like a chess player, the robot learned a "flow," similar to how water flows down a river. It learned the smooth, natural path from a messy table to a clean one. This makes its movements very smooth and fast, rather than jerky or hesitant.

4. Why It's Better Than the Old Way

The researchers tested their robot against the old "one-tool" robots and a robot that tried to guess the goal just by looking at a picture of the finished table.

The Picture Robot: If you showed it a picture of the goal, it got confused in messy rooms because it tried to copy the exact picture, including accidental bumps the human made. It was too rigid.
The Magic Wand Robot: Because it uses simple points (where to push, where to go) and a clear command, it handles messy rooms much better. It ignores the accidental bumps and focuses only on the instruction.

5. The Grand Finale: The Team-Up

The coolest part is how they used this robot in a bigger system. They connected their "Pushing Robot" to a Vision-Language Model (VLM)—basically, an AI that can "see" and "read" like a human.

The Scenario: The VLM looks at a messy table and says, "I need to clean this up."
The Teamwork: The VLM acts as the Manager. It looks at the table, decides, "Okay, let's push these two red cups together so I can grab them both," and then tells the Pushing Robot exactly where to point.
The Result: The robot successfully groups items and clears the table much faster than if it had to pick up every item individually.

Summary

This paper presents a robot that is no longer a "one-trick pony." By using a simple point-and-shoot instruction system (the visual prompt) combined with a smooth learning method (flow matching), the robot can instantly switch between moving, grouping, and separating objects. It's like giving a robot a Swiss Army knife where the blade you need appears automatically based on where you point your finger.

1. Problem Statement

Non-prehensile manipulation, specifically pushing, is a fundamental skill for rearranging objects in cluttered environments. However, existing approaches suffer from two main limitations:

Task-Specificity: Most policies are designed for single, specific tasks (e.g., only bin picking or only object singulation). They lack the versatility to handle diverse scenarios or switch between different pushing behaviors (displacement, grouping, singulation) within a single framework.
Rigid Planning: Many methods rely on pre-defined primitives or multi-step plans that are sensitive to environmental uncertainties (friction, hybrid dynamics) and difficult to generalize to new objects or changing task requirements.
Goal Representation: Traditional goal-conditioned methods often use "goal images," which can be over-specified, difficult to obtain during inference, and prone to over-attention to irrelevant details in cluttered scenes.

The authors propose a Unified Pushing Policy that can generate multimodal pushing actions (displacement, grouping, and singulation) guided by a lightweight visual prompting mechanism, enabling a single model to solve diverse rearrangement tasks efficiently.

2. Methodology

A. Core Framework: Flow Matching

The authors employ Flow Matching, a generative modeling technique based on optimal transport theory.

Advantage over Diffusion: Unlike diffusion models that follow stochastic paths, flow matching learns a deterministic vector field that transforms noise into action trajectories along straight lines. This simplifies the training objective and enables faster inference, which is critical for high-frequency robotic control.
Policy Architecture: The policy is a Diffusion Transformer (DiT) variant. It takes as input:
- Observations ( $O_t$ ): A history of visual data (from wrist and table cameras) and proprioceptive states (end-effector pose).
- Goal Vector ( $g$ ): Encoded from the visual prompt and task specifier.
- Action Chunk ( $A_t$ ): A sequence of future end-effector poses.
Training: The model is trained via Imitation Learning on human demonstration data. It minimizes the Conditional Flow Matching loss to predict the vector field that transforms a noisy action chunk into the ground-truth action chunk.

B. Visual Prompting Mechanism

A key innovation is the Prompting Mechanism, which serves as a lightweight interface for high-level planners to specify tasks without needing full goal images.

Inputs:
1. Task Specifier ( $\psi_{task}$ ): A categorical label indicating the skill type: Displacement, Grouping, or Singulation.
2. Visual Prompts: Pixel coordinates $[u_1, u_2]$ selected on the table camera image.
Semantic Interpretation:
- Displacement: $u_1$ = object to push; $u_2$ = target location.
- Grouping: $u_1$ = source object; $u_2$ = target object to group with.
- Singulation: $u_1$ = object to isolate; the vector $u_2 - u_1$ defines the pushing direction to separate it from clutter.
Classifier-Free Guidance (CFG): To ensure the policy strictly adheres to the prompt, the authors use CFG. During inference, the predicted vector field is a linear combination of the conditional (prompted) and unconditional (null prompt) estimates, pushing the trajectory toward the desired behavior.

C. Goal Predicates

The system defines three specific goal predicates implicitly learned from data:

Displacement: Move a specific object to a target position within a tolerance $\epsilon$ .
Grouping: Bring two objects close together (proximity $\epsilon$ and orientation alignment $\delta$ ) to facilitate multi-object grasping.
Singulation: Isolate a target object from neighbors by maintaining a minimum separation distance $r$ .

3. Key Contributions

Unified Policy: A single flow-matching-based model that encapsulates three distinct non-prehensile skills (displacement, grouping, singulation), allowing for flexible task switching.
Lightweight Prompting: A novel mechanism combining pixel coordinates and task specifiers to guide action generation, offering a more robust and abstract alternative to goal-image conditioning.
Real-World Validation: The method is trained and tested on a real robotic platform (ROBOTIS OpenManipulator-Y), demonstrating superior performance over baselines and successful integration with a Vision-Language Model (VLM) planner.

4. Experimental Results

A. Comparison with Baselines

The authors compared their Unified Policy against:

Single-Task Policies: Three separate models trained only on one task type.
Goal-Image Policy: A flow-matching model conditioned on final-state images.

Results (Success Rate on 20 trials):

Task	Single-Task	Goal-Image	Unified (Ours)
Displacement	75%	60%	85%
Grouping	60%	60%	70%
Singulation	70%	30%	65%

Findings: The unified policy outperformed single-task models in displacement and grouping (suggesting positive transfer learning) and significantly outperformed the goal-image baseline, especially in singulation (65% vs. 30%). The goal-image baseline struggled in cluttered scenes due to "over-attention" to accidental object movements in the goal image.

B. Generalization to Unseen Objects

The policy was tested on novel rectangular objects slightly larger than training objects.
Result: Success rates were maintained or improved (e.g., Singulation rose from 65% to 80%). The larger object size reduced "clipping" errors, proving the model learns generalizable pushing dynamics rather than overfitting to specific object geometries.

C. Integration with VLM Planning

The policy was integrated as a low-level primitive into a VLM-driven planner (Qwen3-VL) for a table-cleaning task.
Workflow: The VLM identifies object centroids and generates a sequence of "Group" or "Grasp" commands. The pushing policy executes the "Group" commands.
Outcome: The system achieved a 50% task completion rate over 10 trials, successfully grouping objects to enable multi-object grasping, thereby reducing the total number of pick-and-place operations required.

5. Significance and Future Work

Versatility: This work demonstrates that a single, unified model can replace multiple task-specific policies, simplifying the deployment of robotic manipulation systems.
Efficiency: The visual prompting mechanism is computationally lightweight compared to querying large foundation models for every action, making it suitable for real-time control.
Scalability: The ability to switch between grouping and singulation allows the robot to adapt its strategy based on scene complexity (e.g., grouping sparse objects vs. singulating cluttered ones).

Limitations & Future Directions:

Friction Dynamics: Failures often occur due to unexpected friction or center-of-mass shifts causing the gripper to lose contact. Future work aims to incorporate world modeling to better capture these dynamics.
VLM Limitations: Current integration relies on off-the-shelf VLMs which have spatial reasoning limitations (e.g., selecting edge points for prompts). Future work will explore more robust planning frameworks.
View Invariance: The current prompting relies on a fixed camera view; future research will investigate view-invariant prompt representations.