💻 computer science

PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration

PhysCaP is a novel agent that enhances robotic manipulation by integrating a training-free, physics-informed exploration layer into code-as-policy frameworks, enabling efficient, targeted information-seeking to infer latent object properties like mass and stiffness through a dual-agent planning and prioritization system.

Original authors: Chen-Yu Lin, Jing-Wen Chen, Hsueh-En Chang, Hung-An Chen, Sheng-Hsun Chang, Chi-Pin Huang, Fu-En Yang, Min-Hung Chen, Yi-Ting Chen, Yu-Chiang Frank Wang, Shao-Hua Sun

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chen-Yu Lin, Jing-Wen Chen, Hsueh-En Chang, Hung-An Chen, Sheng-Hsun Chang, Chi-Pin Huang, Fu-En Yang, Min-Hung Chen, Yi-Ting Chen, Yu-Chiang Frank Wang, Shao-Hua Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the visible world. If a task involves moving a cup from a table to a sink, a modern robot can often see the cup, plan a path, and execute the move with impressive grace. This success relies on vision-language-action policies, systems that learn by watching humans perform tasks and then mimicking those movements. However, these systems operate on a fundamental limitation: they see only what is on the surface. They cannot feel the weight of a can to know if it is empty, nor can they sense the firmness of a fruit to know if it is ripe. In the real world, many tasks depend on these hidden physical properties. A human cleaning a kitchen might shake a soda can to hear if it is empty or squeeze an avocado to check its ripeness, using touch and sound to fill in the gaps that vision leaves behind. Without this ability to actively seek out hidden information, a robot remains blind to the very details that determine whether a task succeeds or fails.

Researchers at National Taiwan University and NVIDIA have developed a new approach to bridge this gap, creating a system they call PhysCaP. This system teaches a robot to act like a curious human, combining the ability to write its own instructions with a new capacity for physical exploration. Instead of just watching and copying, the robot is designed to ask questions of the physical world. When faced with a table of objects where the answer is hidden, the robot does not guess. It decides to interact, lifting or squeezing items to gather the specific data it needs. The system is built on a "code-as-policy" framework, meaning the robot generates executable computer code to control its movements, allowing it to reason through complex steps much like a human planning a sequence of actions. What makes PhysCaP unique is its addition of a physics-informed exploration layer. This layer allows the robot to estimate properties like mass and stiffness using only the feedback from its own motors and joints, without needing special sensors like touch-sensitive skin or scales.

The core of this new system is a dual-agent design that manages the delicate balance between acting too soon and wasting time. One agent, the Planner, looks at the scene and decides if the robot has enough information to finish the job. If the information is missing, the Planner creates a list of potential actions to find it. A second agent, the Prioritizer, then reviews this list and ranks the actions based on visual clues. For example, if the task is to find an empty soda can among four, the Prioritizer notices that two cans are sealed and two have straws sticking out. It logically deduces that the sealed cans are likely full and prioritizes checking the open ones first. This prevents the robot from wasting energy on objects that are unlikely to hold the answer. Once the robot gathers enough evidence, the system stops exploring and executes the final task.

To test this idea, the researchers set up three distinct challenges on a real-world table. In one task, a blue cube was hidden under one of three cups, but one cup was too small to hold the cube. A robot relying only on vision might lift every cup, but PhysCaP used its reasoning to see the size difference and skipped the impossible cup, lifting only the viable candidates. In another task, the robot had to find a single empty soda can among four. Using its ability to measure weight through motor feedback, it successfully identified the empty can. In the third task, it had to pick a ripe avocado from a group of green and dark ones. By squeezing the dark avocados to measure their firmness, it selected the ripe one while ignoring the hard, unripe fruit. In all these scenarios, the system succeeded where other methods failed. Robots that relied only on vision could not solve the tasks because they could not see the hidden properties. Robots that could measure properties but lacked the reasoning to prioritize them ended up checking every single object, taking much longer and using more energy.

The results showed that PhysCaP could complete these tasks with high success rates while interacting with far fewer objects than its competitors. In the real-world tests, the system found the hidden cube, the empty can, and the ripe avocado with significantly fewer physical touches and in less time than systems that checked everything indiscriminately. The researchers also ran simulations to see how the system would perform in a digital environment, and the results held true: the system that could reason about what to measure and when to stop outperformed models that simply tried to guess or check everything. The study confirms that for robots to truly operate in the messy, unpredictable real world, they must be able to actively seek out the physical truths that vision alone cannot reveal. By giving robots the ability to ask the right questions and listen to the answers the physical world provides, this research moves us closer to machines that can handle the subtle, tactile complexities of human life.

Technical Summary: PhysCaP – Grounding Code-as-Policy Agent with Physics-Informed Exploration

Problem Statement

Recent Vision-Language-Action (VLA) policies have demonstrated strong performance in imitating demonstrations for complex manipulation tasks. However, they primarily rely on passive observation, failing to infer latent physical properties (e.g., object mass, stiffness, or internal state) that are critical for tasks involving hidden states. While reinforcement learning can theoretically discover information-seeking behaviors, it often suffers from sample inefficiency and produces policies that are difficult to interpret. Existing Code-as-Policy (CaP) frameworks, which decompose perception and reasoning into modular executable components, improve interpretability but remain fundamentally constrained by passive perception. They lack mechanisms to actively acquire missing physical information through interaction, leading to failures in partially observable environments or inefficient "over-exploration" where agents interact with all objects indiscriminately.

Methodology

The authors propose PhysCaP, a Physics-Informed Code-as-Policy agent designed for active perception in robotic manipulation. The framework integrates three core components to enable efficient, targeted exploration:

1. Training-Free Physical Property Extraction (PhysX) Modules

PhysCaP introduces modules that estimate latent physical properties directly from robot proprioception (joint torques and motor effort) without requiring additional sensors like tactile skins or force-torque sensors.

Mass Measurement (get_mass): Infers object mass by comparing joint torques during a fixed lift trajectory (15 cm) with and without the object. The system calculates differential torque ( $\Delta\tau$ ) and projects it onto the end-effector Jacobian to estimate mass, accounting for kinematic configuration.
Stiffness Measurement (get_stiffness): Estimates object rigidity by analyzing the relationship between gripper jaw displacement and normalized motor effort. The system performs a "backoff" test to verify true contact and measures deformation at a target effort level, mapping the result to a discrete stiffness scale (1–5).

2. Dual-Agent Exploration Architecture

To balance the cost of interaction with the efficiency of information gain, PhysCaP employs a two-stage reasoning process:

Planner Agent: Evaluates whether current visual observations are sufficient to execute the task. If physical properties are hidden, it generates a list of exploration candidates (objects to interact with) and defines the required information (e.g., "weigh the green can"). It also serves as a dynamic stopping criterion, terminating exploration once sufficient evidence is gathered.
Prioritizer Agent: Refines the Planner's candidate list using visual heuristics. It filters out implausible interactions (e.g., weighing a sealed can known to be full) and ranks remaining candidates by a priority score reflecting task relevance. This prevents "over-exploration" by ensuring the robot interacts with the most informative objects first.

3. Code-as-Policy Execution

A coding agent translates the prioritized exploration plan into executable Python code. This code invokes the low-level control APIs (e.g., goto_pose, get_mass) to perform physical interactions. The system operates in a closed loop: after each interaction, new physical data is integrated into the agent's belief state, and the Planner re-evaluates the need for further exploration.

Key Contributions

Physics-Informed Exploration Layer: The introduction of a framework that augments Code-as-Policy agents with active physical interaction, enabling the inference of latent states (mass, stiffness) without external sensing hardware.
Training-Free PhysX Modules: Novel algorithms for estimating object mass and stiffness using only standard robotic proprioception, validated across a range of objects.
Dual-Agent Design: A specific architectural separation between a Planner (determining what and when to explore) and a Prioritizer (determining which object to explore first based on visual cues). This design addresses the trade-off between under-exploration (failure due to lack of info) and over-exploration (inefficiency).
Empirical Validation: Comprehensive evaluation on real-world tabletop tasks (finding hidden cubes, identifying empty cans, selecting ripe avocados) and a simulated environment (LIBERO).

Results

Experiments were conducted on a 7-DoF AgileX PiPER robot arm across three tasks requiring hidden-state estimation:

Task Success Rate (SR): PhysCaP achieved high success rates (80–90% in real-world tasks, 78% in simulation), significantly outperforming passive VLA baselines (which failed due to hidden properties) and basic interactive baselines.
Efficiency (Interactions & Time): PhysCaP required the fewest physical interactions (e.g., 1.33 interactions on average for the "Find Blue Cube" task vs. 3 for baselines) and the shortest execution time.
Comparison to Baselines:
- CaP (Passive): Failed on tasks requiring physical inference.
- CaP+PhysX (Basic Interactive): Succeeded but performed exhaustive exploration (measuring all objects), resulting in high interaction costs and time.
- CaP+PhysX+Planner: Improved efficiency by stopping early but still lacked prioritization, leading to suboptimal interaction sequences.
- PhysCaP: Achieved optimal performance by combining early stopping with visual-heuristic prioritization.
Ablation Studies: Validated that the dual-agent design (separating planning and prioritization) is superior to a merged single-agent approach, which tended to generate unstructured, exhaustive plans. The PhysX modules were shown to accurately estimate mass and stiffness, with an oracle test (using ground-truth mass) achieving 100% success on the "Identify Empty Can" task.

Significance and Claims

The paper claims that PhysCaP represents a significant step toward generalizable robot learning in partially observable environments. By grounding high-level reasoning in explicit physical interaction, the framework allows robots to actively reduce uncertainty rather than relying solely on passive visual cues. The authors emphasize that their approach enables efficient task completion with minimal interaction costs, addressing a fundamental limitation of current VLA models. The work demonstrates that integrating sensor-free proprioceptive estimation with structured, code-based reasoning allows agents to solve manipulation tasks that are otherwise intractable for purely visual or basic interactive systems. The framework is presented as model-agnostic, capable of functioning with various foundation models, and extensible to other physical property extraction modules.