💻 computer science

Action with Visual Primitives

The paper introduces AVP (Action with Visual Primitives), an end-to-end architecture that decouples instruction comprehension and spatial understanding from motor control by having a Vision-Language Model emit visual-primitive tokens to condition a flow-matching action expert, thereby significantly improving success rates, data efficiency, and generalization in robotic pick-and-place tasks compared to existing methods like pi_0.5.

Original authors: Weilong Guo, Yuchen Wang, Renping Zhou, Yunfeng Zhang, Rui Fang, Yue Meng, Wenda Xu, Yuan He, Gao Huang

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weilong Guo, Yuchen Wang, Renping Zhou, Yunfeng Zhang, Rui Fang, Yue Meng, Wenda Xu, Yuan He, Gao Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to play a complex game of Chinese Chess or stack dominoes perfectly. You give it a command: "Move the red piece to the corner."

In most current robot brains (called VLA models), the robot tries to do everything at once. It has to understand your words, figure out exactly where the red piece is on the board, and then calculate the precise muscle movements to grab it—all in a single, split-second thought. It's like asking a person to read a map, drive the car, and park it, all while holding a conversation, without ever stopping to think. This often leads to confusion, especially if the board looks slightly different or the pieces are in a new spot.

The authors of this paper, AVP (Action with Visual Primitives), propose a smarter way to split the work. They treat the robot's brain like a team with two distinct roles: a Strategist and a Doer.

The New Teamwork: Strategist vs. Doer

1. The Strategist (The Vision-Language Model)
Think of this part as the "eyes and brain." Its only job is to look at the scene and the instruction, then decide what needs to be done and where.

Instead of just thinking in words, the Strategist draws a quick, invisible sketch on the image. It might draw a little box around the chess piece or put a dot on the target square.
These drawings are called "Visual Primitives." They are simple, clear markers that say, "Hey, focus here."

2. The Doer (The Action Expert)
Think of this part as the "hands." It doesn't need to worry about understanding the rules of chess or the meaning of your words.

It simply looks at the Strategist's sketch (the Visual Primitives) and says, "Okay, I see the box. I will move my arm to grab whatever is inside that box."
Because the "what" and "where" are already solved by the Strategist, the Doer can focus 100% of its energy on the physical movement.

Why This Works Better

The paper argues that the old way forces the "Doer" to re-learn how to see and understand the world every time it moves. The new AVP method lets the "Strategist" use its pre-trained knowledge to handle the thinking, and the "Doer" just follows the visual clues.

The "Zero Annotation" Superpower
Usually, to teach a robot to draw these boxes, humans have to manually label thousands of pictures (e.g., "This is the chess piece"). That is slow and expensive.

AVP's Trick: The robot already knows where its hand is moving (because it controls its own joints). The system automatically turns the robot's own hand movements into the "Visual Primitives" (the boxes and dots) during training.
Analogy: It's like a dance instructor who doesn't need to draw arrows on the floor for the student. The instructor just watches the student's feet and automatically highlights the steps the student is already taking. No extra drawing required.

The Results: Real-World Proof

The team tested this on a real robot with two arms in three challenging scenarios:

Chinese Chess: Moving pieces on a crowded board with many similar-looking items.
Dominoes: Placing dominoes with very specific angles.
General Pick-and-Place: Picking up random objects of different shapes.

The Scoreboard:

Compared to the previous best method (called π0.5), AVP improved the success rate by 27.61%.
Generalization: If the robot was trained on one type of board but tested on a completely different one (or with different objects), AVP still worked well. The old method often failed because it got confused by the new look.
Speed: While some older methods that used outside tools to find objects were slow (taking 37 seconds for a command), AVP was fast (0.27 seconds), making it practical for real-time use.

In a Nutshell

AVP is a robot control system that separates thinking from doing. It uses a "Strategist" to draw simple visual targets (primitives) based on instructions, and a "Doer" to follow those targets. This division of labor makes the robot much better at handling new tasks, new objects, and crowded environments without needing humans to manually teach it every single detail.

Technical Summary: Action with Visual Primitives (AVP)

Problem Statement

Current Vision-Language-Action (VLA) architectures typically map language instructions and visual observations to robot actions in a single forward pass. While conceptually simple, this design entangles instruction comprehension, spatial scene understanding, and motor control within a single learning objective. Consequently, the "action expert" (the component responsible for generating motor commands) is forced to implicitly relearn cognitive and perceptual capabilities that are already present in the pretrained Vision-Language Model (VLM). This coupling limits learning efficiency and generalization, particularly when objects, layouts, or environments change at test time.

Existing attempts to address this via intermediate representations face specific trade-offs:

Planning-centric approaches (e.g., $\pi0.5$ ) use linguistic subtasks but struggle with fine-grained spatial distinctions in dense scenes.
World-action models (e.g., $\pi0.7$ ) use predicted future frames, requiring the action expert to re-discover task-relevant pixels.
Visual-prompt methods (e.g., Point-VLA, VP-VLA) rely on external models (like SAM or Grounding DINO) to generate spatial markers. This creates multi-stage pipelines that introduce inference latency and risk error propagation from upstream localization failures.

Methodology: AVP Architecture

The authors propose AVP (Action with Visual Primitives), an end-to-end VLA architecture that redefines the interface between the VLM and the action expert. Instead of an implicit, unstructured boundary, AVP employs a visual-primitive-centric communication protocol.

Core Components

Pretrained VLM & Autoregressive Decoder: Given a language instruction ( $l$ ) and multi-view observations ( $o_t$ ), the VLM parses the current subtask. An autoregressive decoder then predicts a discretized visual primitive ( $p_t$ ) representing the next-stage target (e.g., a point, bounding box, or trajectory).
Projection to Visual Space: The predicted primitive is projected from the language space into the visual token space ( $z^{vp}_t$ ). These tokens are overlaid on the original observation to create an augmented multimodal representation ( $z^{aug}_t$ ).
Flow-Matching Action Expert: The action expert receives the augmented tokens and the robot state ( $s_t$ ) to predict a horizon of future actions ( $a_{t:t+h}$ ).

Training and Supervision

End-to-End Optimization: Unlike cascaded approaches, AVP internalizes the visual-primitive interface. The primitive generation and action prediction are optimized jointly.
Supervision Source: The visual-primitive decoder is supervised using an auxiliary loss ( $L_{vp}$ ). Crucially, the ground-truth visual primitives ( $y^{vp}_t$ ) are derived directly from end-effector kinematics (robot proprioception), eliminating the need for manual annotation or external perception models.
Objective: The total loss is a weighted sum of the action prediction loss and the primitive supervision loss: $L = L_{act} + \lambda L_{vp}$ .

Key Contributions

Reconceptualized Responsibility Division: The paper proposes a clear division of labor where the VLM reasons about what to do and where (via visual primitives), while the action expert focuses solely on how to execute.
Visual Primitives as an Explicit Interface: AVP introduces spatially grounded tokens produced internally by the VLM as a communication medium, replacing the implicit boundaries of current VLAs.
End-to-End Implementation: The system implements this interface without external detectors or segmenters, using kinematics-derived supervision to train the primitive generator.
Scalability: The method offers a path to scaling via low-cost training data (kinematics provide automatic labels) and a compact form that extends easily to memory-augmented or longer-horizon tasks.

Experimental Results

The authors evaluated AVP on a dual-arm AgileX Piper platform across three real-world tasks: Chinese chess manipulation, domino placement, and general object pick-and-place.

Performance vs. Baseline: Compared to the $\pi0.5$ $π 0.5$ baseline, AVP achieved an overall 27.61% improvement in manipulation success rate across tasks.
- Chinese Chess: Improved average success rate from 62.67% ( $\pi0.5$ ) to 90.28% (AVP).
- Domino Placement: Improved average success rate from 81.94% to 88.19%, with significant gains in orientation success.
- General Pick-and-Place: Improved average success rate from 64.96% to 86.18%.
Generalization:
- Spatial Compositional Generalization: On unseen direct transitions (A $\to$ B) not seen during training (which only showed A $\to$ C $\to$ B), $\pi0.5$ failed completely (0% success), while AVP achieved high success rates (e.g., 100% on chessboard direct transitions).
- Zero-Shot Object Transfer: A model trained solely on Chinese chess data successfully generalized to unseen objects and backgrounds in a zero-shot setting, whereas the baseline degraded significantly under distribution shifts.
Efficiency: AVP demonstrated lower inference latency (0.27s) compared to methods relying on external LLM APIs (e.g., Point-VLA variants taking ~37s).

Significance and Claims

The paper claims that AVP demonstrates the efficacy of decoupling target inference from motion execution. By offloading complex spatial reasoning to the VLM and providing the action expert with explicit, visually grounded instructions, the system achieves:

Improved Data Efficiency: Learning is more efficient because the action expert does not need to relearn spatial grounding.
Enhanced Generalization: The explicit visual interface allows the policy to recombine learned motion patterns for novel transitions and objects.
Reduced Annotation Overhead: The reliance on kinematics for primitive supervision avoids the cost and fragility of human labeling or external detector dependencies.

The authors acknowledge limitations, noting that the sequential two-stage inference (primitive prediction followed by action generation) introduces slightly higher runtime latency than single-pass policies, and the system remains sensitive to precise hand-eye calibration. However, the results suggest that visual primitives serve as a robust and scalable interface for generalist robotic manipulation.