Technical Summary: Agentic AI for Robot Control: Flexible but still Fragile

Problem Statement

Bridging abstract plans generated by modern planning systems to real-time physical robot control remains a significant challenge. While abstract plans are readily produced, robustly maintaining and updating the beliefs required for plan-following under conditions of partial observability, sensor noise, stale state estimates, and exogenous dynamics is difficult. Existing approaches often struggle to detect, interpret, and respond to unexpected events while maintaining plan consistency and safety. Furthermore, end-to-end approaches that map language and raw perception directly to low-level control lack interpretability and verifiability. This paper addresses the need for a control architecture that leverages the reasoning capabilities of Large Language Models (LLMs) while operating over a higher-level interface of interpretable robot skills and verifiable state predicates.

Methodology

The authors propose an agentic control system centered on a single planner and executor loop that interleaves natural language plan construction, tool-grounded execution, and execution monitoring. The system is deployed on two physical platforms: the Mobipick (indoor mobile manipulator) and Valdemar (outdoor agricultural robot).

Architecture

The workflow is triggered by a speech recognition module and routed by a lightweight LLM (gpt-5-mini) to either a chatbot or the primary Planner and Executor agent (using the o3 reasoning model).

• Planner and Executor: This agent uses a structured prompt to reason over a semantic environment snapshot (a sensor-driven state representation) and a constrained action catalogue. It operates via a closed loop:
  1. Reflection: Produces a structured high-level plan.
  2. State Acquisition: Queries the current semantic state.
  3. Action: Invokes a single high-level tool (e.g., navigate, pick, place).
  4. Event Check: Queries for discrete, text-based exogenous events.
• Goal Completion Critic: A secondary LLM (o4-mini) inspects the execution log to distinguish between genuine physical attempts and "verbal only" failure modes (where the model explains a task without acting). It can request continuation or halt execution if the goal is infeasible or blocked.
• Tool Interface: The agent interacts with the robot via a minimal set of tools: reflect (introspection), act (executing one high-level action), get semantic environment representation snapshot, and check events.

Prompt Engineering

The system relies on a modular prompt structure containing:

• Domain Model: Defines embodiment, sensing semantics, and operational constraints (e.g., "retract arm before navigation").
• State Information: Provides examples of the semantic state snapshot, emphasizing that knowledge is incomplete and time-varying.
• Operational Instructions: Rules governing execution behavior, safety invariants, and event handling.
• Affordances and Heuristics: Task-relevant object properties and risk-reducing preferences (e.g., minimizing actions, delaying placement).
• Action Catalogue: A defined list of permitted environment-changing operations.
• Failure Patterns: Hand-curated records of past failures to bias the agent against repeating specific errors.

Execution Pipeline

The system assumes access to a robot action API exposing pre-programmed, parametrized atomic skills (e.g., ROS-based navigation, grasp synthesis). The LLM does not learn low-level control; instead, it orchestrates existing skills. Execution feedback (success/failure/error traces) is returned as text to the agent, enabling iterative recovery and replanning without hard-coded logic.

Key Contributions

1. Real-World Agentic Integration: A demonstration of a planner-executor translating natural language goals into tool calls over existing robot stacks, featuring iterative recovery under partial observability and intermittent failures.
2. Architecture-Level Design: A separation of geometric/spatial grounding (handled by the underlying robot stack) from language-based deliberation. The LLM reasons over structured state snapshots and parametrized skills rather than raw geometry.
3. Reproducible Engineering Specification: A concrete definition of prompt structure and tool contracts (domain models, affordances, event checking, recovery patterns), accompanied by a qualitative analysis of observed behaviors and fragility.

Results and Qualitative Validation

The system was validated on the Mobipick and Valdemar platforms through seven experiments:

• Mobipick (Indoor): The system successfully executed nominal tasks (e.g., "put multimeter in box and bring to table 2") and handled ambiguous commands (e.g., selecting a tool for "screwing"). It demonstrated the ability to incorporate user interventions (emergency stops, preference changes) and reason about constraints (e.g., perceiving a table surface before placing an object on it to avoid collisions).
• Valdemar (Outdoor): The system performed autonomous waypoint scanning and battery management. It successfully interrupted missions to recharge when battery levels dropped and refuted impossible commands (e.g., "Fly to Paris") using domain knowledge.
• Long-Horizon Stress Test: The system managed complex, multi-step tasks involving container usage and cluttered workspaces, recovering from execution-level interruptions.

Observed Limitations (Fragility)

Despite flexibility, the system exhibited substantial fragility:

• Non-deterministic Behavior: Suboptimal planning (e.g., inefficient search paths) and inconsistent event checking (e.g., rapid, repeated invocations).
• Instruction-Following Errors: Occasional "verbal only" failures where the model explains a solution without executing actions.
• Prompt Sensitivity: High sensitivity to prompt specification; safety constraints often needed to be redundantly stated across multiple prompt sections to be effective.
• Latency and Cost: Reliance on remote model inference introduces non-negligible latency and monetary cost per query.
• Asynchrony: The system uses polling for event detection, meaning long-running actions cannot be preempted immediately if an event occurs mid-execution.

Significance and Claims

The paper posits that while current LLM-based agentic systems are flexible—allowing transfer to different robot domains primarily through prompt updates and API re-binding—they remain fragile in real-world deployment.

The significance of this work lies in:

1. Demonstrating Feasibility: Showing that a single planner-executor loop can handle heterogeneous tasks (indoor manipulation vs. outdoor navigation) with minimal architectural changes.
2. Transparency: The system's ability to verbalize its reasoning and state provides a level of interpretability for non-expert operators that end-to-end models lack.
3. Contextual Utility: Leveraging the large context windows of modern LLMs allows for maintaining rich task context and execution traces within a single loop, enabling long-horizon tasks that were previously less feasible.
4. Safety via Abstraction: By restricting the LLM to high-level decision-making and valid API calls, the system ensures safe behavior even if the model hallucinates, as it cannot directly execute invalid low-level commands.

The authors conclude that while robust domain modeling remains labor-intensive and prompt sensitivity is a hurdle, this agentic approach offers a viable path for integrating commonsense reasoning into physical robot control, provided that safety constraints are explicitly and redundantly encoded.