Technical Summary: VIA: Visual Interface Agent for Robot Control

Problem Statement

General-purpose robot manipulation requires a synthesis of visual understanding, physical reasoning, long-term planning, and precise closed-loop control. While Foundation Models (FMs) have demonstrated remarkable capabilities in vision and reasoning, current approaches to leveraging them for robotics face significant limitations:

1. Vision-Language-Action (VLA) Models: These typically involve fine-tuning FMs on limited robot data to output low-level actions. However, due to compute and data constraints, these models are often orders of magnitude smaller than frontier FMs, limiting their general reasoning and planning capabilities.
2. Code-as-Policies (CaP): This paradigm prompts FMs to generate code that calls perception APIs and hand-crafted skill primitives. While it leverages FM coding abilities, the system is often bottlenecked by the quality of human-crafted abstractions. If these primitives are replaced with lower-level ones, success rates degrade sharply.

The core problem addressed is how to utilize the full reasoning and planning capabilities of frontier agents for robot control without the need for robot-specific fine-tuning or reliance on brittle, task-specific abstractions.

Methodology: VIA Framework

The authors propose VIA (Visual Interface Agent for robot control), a framework that recasts robot control as an agentic visual tool-use task, analogous to how agents operate computer software.

Core Concept

Instead of training a model to output joint torques or end-effector velocities directly, VIA treats the robot environment as a browser-based 3D application. An off-the-shelf FM-powered agent interacts with this interface using a set of human-legible tools. The agent operates in a closed-loop observe-act cycle:

1. Observe: The agent takes screenshots of a 3D interface reconstructed from RGB-D cameras. It has no access to privileged state information (e.g., raw point clouds or internal simulator states).
2. Reason & Plan: The agent analyzes the visual input to determine the next step.
3. Act: The agent issues commands via Model Context Protocol (MCP) tools to manipulate a virtual "target gripper" within the interface.
4. Execute & Verify: Once a target pose (waypoint) is set, the agent invokes a controller to move the real robot to that pose. The agent then observes the outcome via a new screenshot and re-plans if necessary.

The Visual Interface

The interface renders a 3D point cloud of the scene alongside third-person and wrist camera feeds. It mimics 3D design software (e.g., Blender), allowing the agent to:

• Manipulate a blue virtual target gripper.
• Change the viewport (orbit, pan, zoom) to disambiguate the scene.
• Set 6-DoF target poses and binary open/close states.

MCP Tools

The interface is wrapped in a minimal set of MCP tools designed for agent ergonomics:

• Observation: screenshot, hover (to verify 3D coordinates), gripper_get_pose, camera_get_pose.
• Action: gripper_teleport_via_click, gripper_drag, gripper_translate, gripper_rotate, gripper_toggle. These tools modify the virtual target gripper on the UI.
• Execution: execute_waypoint (moves the real robot to the target pose) and end_episode.
• Design Principles: The tools are minimalist (wrapping human operations with little abstraction) and ergonomic (providing discrete, angle-based inputs rather than continuous drag operations to suit screenshot-based observation).

Agentic Loop

The agent (e.g., Claude Code or Codex) receives a system prompt describing the interface and a task goal. It decomposes the task into a sequence of waypoints. Crucially, the agent performs closed-loop error recovery: if an execution fails or the outcome is unexpected, the agent diagnoses the issue from the new screenshot and adjusts its plan without human intervention.

Key Contributions

1. Recasting Robot Control as Agentic Tool Use: The paper demonstrates that robot manipulation can be solved by treating the robot as another application an agent can operate, bypassing the need for robot-specific fine-tuning.
2. Zero-Shot Generalization: VIA enables state-of-the-art agents to solve diverse manipulation tasks zero-shot, relying solely on their pre-trained general reasoning and visual understanding capabilities.
3. Scalability: Because VIA uses frontier models unmodified, improvements in agent perception, reasoning, or tool use transfer directly to robot control capabilities.
4. Text-Based Demonstrations: The framework allows for "demonstrations" to be provided as plain text waypoint lists, offering a low-cost alternative to teleoperation for improving performance.

Experimental Results

The framework was evaluated on six tabletop manipulation tasks (Stack, Turn on stove, Open drawer, Put bowl on plate, Rainbow assembly, T-block) using Claude Code (CC) and Codex with various model tiers (Opus, Fable, GPT-5.5, GPT-5.6-Sol).

• Zero-Shot Performance: With minimal prompts, VIA achieved success rates ranging from 60% (Codex-5.5) to 88% (CC-Fable) across tasks.
• High-Performance Configurations:
  • LIBERO-Goal Tasks: CC-Fable achieved 96.7% success on "Turn on stove" and "Open drawer" tasks.
  • Long-Horizon Assembly: CC-Fable achieved 100% success on the "Rainbow" task (arranging seven blocks), a task requiring complex long-horizon planning.
• Impact of Text Demonstrations: Providing a detailed prompt with an example waypoint list improved CC-Opus performance on LIBERO-Goal tasks from 77% to 100%.
• Scaling: Performance consistently improved with the strength of the underlying model (e.g., upgrading from CC-Opus to CC-Fable increased overall success from 70% to 88%).
• Cost: The estimated API cost per successful episode was a few dollars (e.g., ~$4.1 for Codex-5.6-Sol, ~$9.5 for CC-Opus).

Significance and Claims

The paper argues that frontier agents already possess the skills necessary for robot control—specifically general perception, spatial reasoning, and planning—provided they are given the right interface.

• "Secretly a Robot-Control Agent": The authors posit that a coding or computer-use agent is, in a sense, a robot-control agent. By presenting the robot through a visual interface, these agents can transfer their existing competencies directly to physical manipulation.
• Economic Viability: The framework demonstrates that robot control can become an economically valuable agentic task that benefits directly from modern model scaling without the need for expensive, task-specific data collection or fine-tuning.
• Limitations: The authors acknowledge that VIA currently requires frontier models, resulting in slow inference and higher costs, making it suitable for quasi-static tasks rather than dynamic ones (e.g., catching a ball). They also note that precision-demanding tasks (like the T-block) remain challenging due to the limitations of the underlying controller and the discrete nature of the interface.

In conclusion, VIA opens a new pathway for leveraging large-scale foundation models in robotics, suggesting that the future of generalist robot policies may lie not in training smaller, specialized models, but in connecting the most capable general agents to physical systems through intuitive visual interfaces.