Technical Summary: P3: Toward Versatile Embodied Agents

Problem Statement

Current embodied agents face significant barriers to versatile real-world deployment due to three primary limitations:

1. Passive Environmental Perception: Existing systems rely heavily on feedback from tool agents (e.g., APIs, external modules) to infer environmental changes. This approach fails to detect real-time dynamics caused by external factors (e.g., human movement, other agents) or when tools lack bidirectional communication capabilities (common in IoT and basic mechanical devices).
2. Tool Dependency and Rigidity: The reliance on predefined toolkits with strict feedback mechanisms limits the set of available operations. Many real-world edge devices cannot provide the necessary feedback due to hardware constraints, security restrictions, or lack of communication protocols.
3. Inflexible Multi-Task Planning: Prior task scheduling often follows a rigid first-in-first-out (FIFO) order, lacking the ability to manage task dependencies, user priorities, or dynamic interruptions. This leads to suboptimal performance in complex environments where urgent tasks (e.g., safety checks) must override routine activities (e.g., tidying).

Methodology: The P3 Framework

The authors propose P3, a unified framework integrating Perceive, Plug, and Plan capabilities to address these challenges. The architecture centers on decoupling perception from specific tool agents and utilizing Large Language Models (LLMs) for dynamic scheduling.

1. Active Omni-Task Perception

Instead of relying on tool feedback, P3 employs a centralized, always-on Perception Module powered by Vision-Language Models (VLMs).

• Mechanism: The module continuously analyzes video streams to detect scene changes, infer causal events, and update a Scene Graph representing spatial relations.
• Task Generation: It proactively identifies potential tasks (e.g., "door left open," "debris on floor") based on dynamic context, converting visual cues into structured task proposals.
• Memory Management: To handle context limits, the system uses hierarchical memory buffers (short-term low-compression and long-term high-compression) and on-the-fly summarization to feed only relevant events to the planner.

2. Plug-Any-Tool Integration

P3 introduces an open-ended tool mechanism that removes the dependency on bidirectional feedback.

• Decoupling: Tool agents (e.g., IoT controllers, web agents, navigation, manipulation) are integrated without requiring confirmation signals.
• Flexibility: Users can register custom tools simply by defining their function and call interface. The system validates commands for safety but does not wait for execution feedback, relying instead on the Perception Module to observe the resulting environmental changes.
• Scope: This enables seamless interaction with diverse devices, including those with one-way communication or no standard interfaces.

3. LLM-Guided Multi-Task Planning

A dynamic Task Planner manages a mix of three task types:

• Passive Tasks: Triggered by explicit user instructions.
• Active Tasks: Proactively proposed by the Perception Module.
• Scheduled Tasks: Time-triggered tasks registered by the user.
• Execution Logic: The planner evaluates all pending tasks based on priority, dependencies, and user instructions. It acts as an "air-traffic controller," selecting high-priority tasks for execution, managing interruptions, and resuming suspended tasks using a Task Memory that stores state and progress.

4. Memory Architecture

The framework utilizes four complementary memory components to support grounding and consistency:

• Spatial Memory: Stores a scene graph with room names and object relations to resolve linguistic ambiguity.
• Event History: A time-stamped log of changes used for causal reasoning and priority updates.
• Task Memory: Tracks current, pending, and interrupted tasks to enable resumption.
• Robot State: Stores instantaneous constraints (e.g., battery, gripper occupancy) to ensure feasible dispatching.

Key Contributions

1. Omni-task Perception: A unified module that actively detects real-time environmental changes, allowing agents to identify tasks autonomously without passive feedback.
2. Plug-Any-Tool Agent: An architecture that enables seamless integration with a wide range of devices, including those lacking standardized feedback mechanisms, significantly enhancing flexibility.
3. Dynamic Multi-task Planning: A scheduler that jointly processes active, passive, and scheduled tasks, managing dependencies and priorities to handle complex, competing real-world scenarios.
4. Real-world Validation: Extensive deployment on humanoid dual-arm robots (Realman and AgiBot) in laboratory and office settings, demonstrating the framework's ability to bridge the gap between benchmarks and practical application.

Experimental Results

The authors evaluated P3 through an Active Task Perception Benchmark and 11 real-world embodied tasks.

• Perception Benchmark: Using 600 snapshots from first-person videos, the study evaluated various VLMs. QwenVL-max achieved the best balance, with an overall accuracy of 45.85% (33.97% on positive samples and 81.48% on negative samples), proving its suitability for active task proposal.
• Real-world Deployment:
  • Tool Integration: P3 successfully executed tasks involving navigation, manipulation, IoT (e.g., controlling humidifiers), and web agents. It achieved 100% success on several independent tasks (e.g., Task 2: turning off a humidifier; Task 4: weather broadcast).
  • Competing Tasks: In complex scenarios involving interruptions and priority shifts (Tasks 7–11), P3 demonstrated robustness. For instance, in Task 11 (picking up a knocked-over cup while handling other instructions), P3 achieved a 42.86% success rate, significantly outperforming the baseline RoboOS (35.71%), which failed to manage interruptions effectively.
  • Comparison: Unlike RoboOS, which follows a strictly linear, passive execution paradigm, P3 autonomously detected and acted upon environmental changes (e.g., picking up debris without instruction) and successfully managed task resumption after interruptions.

Significance and Claims

The paper claims that P3 represents a significant step toward versatile and scalable embodied agents. By eliminating the dependency on restrictive tool feedback and introducing active perception, the framework allows agents to operate in unstructured, dynamic environments where traditional systems fail. The authors assert that P3 successfully bridges the gap between controlled benchmarks and practical deployment, delivering a system capable of handling dynamic challenges, managing competing priorities, and integrating with diverse hardware without extensive retraining or custom feedback loops. The work also lays a foundation for scalable embodied data collection.