Technical Summary: AxisGuide

Problem Statement

Recent visuomotor manipulation policies, particularly those leveraging large-scale behavior cloning and Vision-Language-Action (VLA) models, have demonstrated strong semantic scene understanding. However, a critical gap remains between semantic comprehension and reliable low-level action execution, especially under distribution shifts. Even in tasks with identical scene layouts, camera viewpoints, and illumination, performance degrades significantly when objects are placed at unseen locations.

The authors argue this failure stems from insufficient action understanding: the policy's inability to interpret the robot's base-frame action coordinate system within the image space. Standard RGB observations and proprioceptive states rarely provide an explicit visual reference anchoring the action coordinate system (e.g., what "+x" or "+z" motion looks like in pixels). Consequently, policies often degenerate into memorizing image-to-vector correspondences rather than learning the geometric relationship between visual observations and the robot's base-frame actions. This leads to poor generalization when the target object shifts, as the model cannot correctly compose new relative end-effector displacements.

Methodology: AxisGuide

To bridge the gap between semantic understanding and action-coordinate interpretation, the authors propose AxisGuide, a lightweight guidance method that explicitly grounds the robot's action coordinate system in RGB observations.

Core Mechanism

AxisGuide augments standard RGB observations with a small set of additional channels that visualize the meaning of action dimensions (+x, +y, +z) in the image space.

1. Projection of Axes: Using known camera intrinsics and extrinsics, along with the current end-effector (EEF) pose, the method projects the robot's base-frame axes onto the 2D image plane.
2. Visual Cues: For each camera view, the method renders three direction vectors originating from the projected EEF location. These vectors correspond to unit translations along the robot base frame's $x$, $y$, and $z$ axes.
3. Representation: These vectors are encoded as a 3-channel image ($A_t \in \mathbb{R}^{3 \times H \times W}$), where the red, green, and blue channels represent the $x$, $y$, and $z$ directions, respectively.
4. Integration: The cue image is concatenated channel-wise with the original RGB image (increasing input channels from 3 to 6 per view) rather than overlaid on top. This preserves the raw appearance distribution while providing a dedicated, disentangled channel for the policy to exploit.
5. Policy Learning: The policy (e.g., Diffusion Policy or SmolVLA) is trained via standard behavior cloning on this augmented observation space. The visual backbone's first convolution layer is simply adjusted to accept the increased channel dimension.

Key Design Choices

The authors identify three critical design choices for effective representation:

• Injection: Channel-wise concatenation is preferred over overlaying, as it prevents occlusion of critical visual details.
• Positioning: Anchoring the cues at the end-effector location (EEF-aligned) is superior to a fixed center-aligned position, as it provides an action-centric reference relevant to the current interaction point.
• Normalization: Normalizing the projected vectors to unit directions removes scale variations caused by depth and camera geometry, allowing the policy to focus on directional semantics.

Key Contributions

1. Problem Identification: The paper identifies and defines the "action coordinate system understanding" gap as a primary cause of poor generalization in visuomotor policies under distribution shifts.
2. AxisGuide Proposal: A simple, effective solution that injects explicit references for the action coordinate system into RGB images without requiring depth sensors, extra supervision, or major architectural changes.
3. Empirical Validation: Demonstration of consistent performance gains and improved generalization across diverse camera configurations (single-view and multi-view) in both simulation (LIBERO benchmark) and real-world environments.

Experimental Results

The authors evaluated AxisGuide against baselines including Diffusion Policy (DP) and SmolVLA across multiple tasks.

• Generalization to Unseen Locations: In a "Pick Up" task with randomized object placements, AxisGuide significantly improved success rates on unseen locations.
  • Simulation: Success rate increased from 52.38% to 65.71% (+13.33%p).
  • Real-World: Success rate increased from 30.12% to 50.00% (+19.88%p).
• Single-Task Performance: In the LIBERO simulation, AxisGuide improved success rates across all tasks (Pick & Place, Drawer, Stove), with the largest gain (+13.34%p) on Pick & Place. In real-world single-view settings, gains were observed on Pick & Place (+10.00%p) and Flip Pot (+13.35%p).
• Multi-Task Performance: When applied to SmolVLA across 10 tasks in the LIBERO suites, AxisGuide consistently outperformed both frozen-backbone and fully-trained baselines, showing a +7.5% gain on the LIBERO-Object suite.
• Comparison with Baselines: AxisGuide outperformed other visual cue methods like TraceVLA (motion traces) and AimBot (reticles), which focus on scene-level spatial relationships but fail to explicitly ground the robot's action coordinate system.
• Robustness: The method demonstrated robustness to calibration errors (extrinsic and intrinsic perturbations), maintaining performance above the baseline even with injected noise.

Significance and Claims

The paper claims that AxisGuide effectively bridges the gap between semantic scene understanding and action execution by making the action coordinate system explicit in pixel space. The authors emphasize that:

• Visuomotor policies struggle to generalize not because they lack semantic understanding, but because they fail to map high-level intents to low-level base-frame actions in novel spatial configurations.
• Explicitly visualizing the action coordinate system allows policies to adapt end-effector motions more reliably when targets shift, leading to robust, transferable generalist policies.
• This approach offers a lightweight, model-agnostic improvement that does not rely on complex architectural changes or depth information, making it applicable to a wide range of existing visuomotor pipelines.

The authors acknowledge limitations, noting that the method requires access to camera parameters (though these are often available or estimable) and that coordinate grounding alone does not resolve all errors, particularly in contact-rich stages like grasping. They suggest that future work could explore scaling these findings to massive, diverse datasets like DROID.