Technical Summary: OmniAct – Advancing Omnimodal Embodied Agents

1. Problem Statement

The paper addresses the fundamental challenge of building persistent embodied agents capable of operating reliably in unstructured, real-world environments over extended temporal horizons. Current embodied systems face three critical limitations when deployed in such demanding scenarios:

1. Lack of Unified Action Space: Existing approaches treat cyber-domain tool invocation (APIs, IoT) and physical-domain manipulation (robotic control, navigation) as separate pipelines. This separation forces brittle, hand-crafted interfaces that fail to handle naturally interleaved instructions requiring joint orchestration of heterogeneous tools.
2. Unbounded Context Accumulation: Agent frameworks that linearly concatenate interaction history suffer from rapid context growth during long-term deployment. This leads to increased memory overhead, the loss of critical historical information, and semantic conflicts between long-term user preferences and recent states.
3. Open-Loop Execution Failures: While Vision-Language-Action (VLA) models have advanced, their inherently open-loop nature prevents autonomous recovery. Undetected physical failures (e.g., grasp slippage, navigation errors) cascade into irreversible faults, as the system lacks mechanisms to verify execution progress or trigger replanning without human intervention.

The core problem is achieving everyday physical autonomy, which requires coordinating heterogeneous cyber-physical tools within single task flows and autonomously recovering from physical failures over extended horizons.

2. Methodology: The OmniAct Framework

The authors propose OmniAct, a hierarchical asynchronous architecture that explicitly decouples planning, memory, and verification into specialized, cooperating modules. Rather than relying on a monolithic end-to-end model, OmniAct integrates three tightly coupled components:

A. Multimodal Semantic Planner

This module instantiates the high-level planning policy ($\pi$). It fuses multimodal inputs (speech, vision, state) to perform:

• Intent Interpretation: Grounding ambiguous user instructions against current tool states and visual observations.
• Task Decomposition: Breaking long-horizon instructions into executable sub-goals.
• Skill Routing: Selecting appropriate skills ($\tau_t$) from a unified inventory ($\mathcal{T}$) that spans both cyber (IoT, Web APIs) and physical (manipulation, navigation) domains.
• Output: A structured JSON specification supporting parallel dispatch of multiple skill calls within a single step.

B. Closed-Loop Execution with Asynchronous Visual Preemption

To address the open-loop limitation of VLA policies, OmniAct employs an asynchronous visual preemption engine:

• Dual-Stream Execution: Discrete cyber commands are verified via deterministic return codes. Continuous physical commands are forwarded to control policies without inherent verification signals.
• Semantic Verification: An asynchronous monitor periodically samples visual frames and submits them to a Vision-Language Model (VLM) to assess execution progress and physical safety at a semantic level (e.g., "Is the object still grasped?", "Is the robot moving toward the goal?").
• Preemption & Replanning: Upon detecting an anomaly, the system immediately halts the VLA executor, packages the failure context, and returns control to the planner for replanning. This converts catastrophic failures into recoverable delays.

C. Adaptive Hierarchical Memory

To manage long-horizon context without linear growth, the framework utilizes a three-level memory hierarchy driven by event-boundary compression:

1. Sensory Memory: A lightweight FIFO queue for recent visual frames, decoupled from the planning context.
2. Episodic Memory: Interaction history is compressed at semantic event boundaries (e.g., sub-task completion). Raw observation sequences are condensed into compact keyframe summaries retaining semantic content and state transitions while discarding procedural redundancy. This ensures context grows proportionally to distinct semantic events rather than raw turns, achieving sub-linear scaling.
3. Reflective Memory: Stores static priors (tool schemas) and dynamically accumulated experience. When failures occur, the system generates structured records of root causes and corrective strategies. These reflections are retrieved during subsequent planning to transform isolated error recovery into cumulative adaptation.

3. Experimental Evaluation

The framework was evaluated across 40 real-world long-horizon tasks on two structurally distinct robotic platforms:

• UR5e: A 6-DoF manipulator for precision tabletop manipulation.
• Keenon: A wheeled mobile robot for indoor navigation.
• Environment: Coordinated with 4 household IoT devices and 12 virtual tool APIs.

Tasks were stratified into three complexity levels:

• Level 1: Physical only (low-level control).
• Level 2: Cyber-physical (IoT coordination).
• Level 3: Full multimodal (Speech, APIs, IoT, and physical execution).

Key Results

• Performance: OmniAct significantly outperformed baselines (Direct Policy, SayCan, Code as Policy, RoboBrain2) across all complexity levels. On Level 3 tasks, it achieved 50.0% (manipulation) and 54.2% (navigation) End-to-End (E2E) success rates, substantially exceeding the best baselines.
• Robustness: The framework demonstrated graceful performance decline as task complexity increased, whereas baselines suffered steep degradation. OmniAct's ability to detect mid-task failures and replan prevented error cascades.
• Scalability & Efficiency:
  • Context Growth: While raw history concatenation grew linearly to over 160k tokens, OmniAct's hierarchical memory maintained a near-flat token consumption profile (2.5k–10k tokens per call) regardless of accumulated history length.
  • Constraint Satisfaction: In long-horizon scenarios (10+ hours, 40k+ tokens), OmniAct achieved an 86.67% constraint satisfaction rate, significantly outperforming sliding window (13.33%) and raw history (66.67%) approaches.
• Generalizability: OmniAct elevated mid-scale open-weight models (e.g., Qwen3-VL-30B-A3B) to match the performance of proprietary models (Gemini-3.1-Pro), demonstrating that architectural orchestration can close the gap between open-weight and proprietary systems without additional training.

4. Significance and Claims

The paper argues that persistent real-world autonomy does not require monolithic model scaling but rather a hierarchical asynchronous architecture with explicit separation of responsibilities.

• Unified Cyber-Physical Control: OmniAct demonstrates that discrete cyber tools and continuous physical control can be unified into a single event-driven loop, enabling seamless coordination of heterogeneous tools without force sensing.
• Bounded Context for Long-Horizon Autonomy: By utilizing event-boundary-driven compression and reflective memory, the framework enables agents to maintain bounded context growth while autonomously recovering from failures and adapting to dynamic environmental shifts.
• Architectural Efficiency over Model Capacity: The results suggest that explicit separation of planning, memory, and verification offers a more viable and resource-efficient path toward persistent embodied deployment. The framework's structured orchestration allows mid-scale models to achieve proprietary-level performance, underscoring that architectural clarity complements model capacity.

The authors conclude that their approach addresses the fundamental gaps in current embodied systems, providing a robust foundation for agents that can operate reliably in unstructured environments over extended temporal horizons.