Technical Summary: Embodied Agents Take Control

Problem Statement

Autonomous embodied agents require a sustained decision-making loop involving perception, action, verification, and self-correction over extended interactions. Current approaches generally fall into two categories: trained policies (where navigation capability is internalized in model weights, e.g., NaVid) and zero-shot workflows (where a frozen general-purpose model is guided by human-authored scaffolding, e.g., NavGPT). In both paradigms, the interaction loop is directed by external code or fixed pipelines, limiting the agent's ability to dynamically manage its state, recover from errors, or decide when to terminate.

This paper investigates a third paradigm: agentic embodied control, where a general-purpose agent holds the interaction loop itself. The core research question is whether a model, equipped only with a minimal interface (a monocular RGB camera and discrete actions), can sustain the high-level control of embodied navigation without navigation-specific training, maps, or explicit memory.

Methodology

The authors propose a minimal-interface probe to test agentic control. They replace the navigation-specific scaffolding of existing zero-shot systems with a general-purpose software-engineering agent harness.

• Interface: The agent receives only a single 512×512 front-facing RGB frame (observe()) and a set of four discrete Habitat primitives (step()): Forward (0.25m), Left (15°), Right (15°), and Stop.
• Constraints: The agent has no access to depth, odometry, pose, maps, waypoints, or collision feedback. It must infer landmarks, progress, and stopping conditions solely from the history of visual observations and action outcomes.
• Setup: The study utilizes the R2R-CE (Room-Across-Room Continuous Environment) val-unseen rand100 subset. The agents are evaluated on three distinct software harnesses originally designed for software engineering (mini-swe-agent, Claude Agent SDK, Codex CLI) and various frontier Vision-Language Models (VLMs).
• Experimental Design: The authors conduct ablation studies across three axes:
  1. Model: Varying the underlying VLM while keeping the harness and interface fixed.
  2. Harness: Comparing different agent frameworks (open-source vs. vendor SDKs) with the same model.
  3. Interface: Comparing the minimal primitive interface against a hybrid setup where a trained waypoint predictor is exposed as an optional tool rather than a forced control path.

Key Contributions

1. Demonstration of Agentic Embodied Control: The paper instantiates agentic control in navigation using unmodified software-engineering harnesses. Despite receiving no navigation-specific training or scaffolding, these agents achieve competitive performance on standard zero-shot benchmarks.
2. Capability Localization: Through single-axis interventions, the authors isolate the source of capability. They find that model choice is the primary driver of success (spanning a 5–72% Success Rate range), while harness differences are largely descriptive. Furthermore, they show that a trained waypoint module, when imposed as a forced control path, can constrain strong models but rescue weak ones; however, when exposed as an optional tool, it allows the strongest agents to compose coarse and fine control, improving performance, speed, and efficiency.
3. Characterization of Limits: The study identifies the gap between short-horizon agentic control and sustained autonomy. It highlights that while reasoning transfers to physical hardware, the lack of body awareness (e.g., collision detection, kinematic clearance) and persistent spatial memory prevents reliable operation on longer horizons.

Results

• Performance: Under the strictly minimal interface, frontier models achieve high success rates on R2R-CE.
  • Opus-5 (default effort): 70.7 ± 3.5% Success Rate (SR).
  • Fable-5 (default effort): 68.3 ± 1.5% SR.
  • Fable-5 (maximum effort): 78% SR.
  • Hybrid Interface (Fable-5): When given the option to use a trained waypoint tool, Fable-5 reaches 76.7 ± 0.6% SR using half the environment steps and less than 25% of the wall time compared to the maximum-effort primitive run.
• Comparison: These results are competitive with engineered zero-shot workflows (e.g., AgenticNav at 55% SR) and recent industrial-scale trained policies (e.g., Qwen-RobotNav at 66–72% SR), despite the minimal interface lacking the extensive scaffolding (maps, memory, search) used by those systems.
• Failure Analysis: Failures are primarily linguistic and spatial rather than perceptual. Common failure modes include binding ambiguous referents to the wrong object instances, stopping based on object proximity rather than the annotated endpoint, and "runaway search" where the agent fails to recover from a wrong branch. A critical finding is self-diagnosis without self-correction: agents often correctly identify a doubt in their reasoning but proceed to commit to the wrong action anyway.
• Physical Deployment: When deployed on a Unitree Go2 quadruped, the agent's reasoning and perception capabilities transferred well (e.g., handling conditional logic and object disambiguation). However, failures occurred due to a lack of body awareness (e.g., turning before the robot's body cleared a doorway) and the absence of a persistent spatial map, leading to counting errors and hallucinations over long routes.

Significance and Claims

The paper claims that agentic control is already competitive in zero-shot navigation, challenging the necessity of heavy, task-specific scaffolding for short-horizon tasks. The results suggest that the capability for embodied decision-making is increasingly residing within the general-purpose model itself.

The authors argue that the future of autonomous embodied agents lies in a bootstrapping cycle:

1. Current scaffolds (maps, waypoint modules, engineered harnesses) generate successful trajectories.
2. These trajectories are used to train better models via supervised fine-tuning or reinforcement learning.
3. Stronger models, in turn, require less scaffolding, eventually simplifying the system architecture.

However, the paper modestly concludes that while agentic systems can produce capable decisions, they are not yet autonomous embodied agents capable of sustained, open-ended operation. The transition requires solving the "embodiment gap": developing harnesses with bounded context management and persistent state, and training models to internalize body awareness and spatial memory. The current work serves as a diagnostic probe, demonstrating that the model can direct the loop, but the surrounding system must evolve to support sustained autonomy.