ALRM: Agentic LLM for Robotic Manipulation

Technical Summary: Agentic LLM for Robotic Manipulation (ALRM)

Problem Statement
While Large Language Models (LLMs) have demonstrated advanced reasoning and planning capabilities, their integration into robotic control pipelines remains constrained by two primary limitations. First, prior LLM-based approaches often lack modular, agentic execution mechanisms, preventing systems from planning, reflecting on outcomes, and revising actions in a closed-loop manner. Second, existing benchmarks for robotic manipulation tasks predominantly focus on low-level control, failing to systematically evaluate multistep reasoning or the robustness of agents against linguistic variations in task instructions.

Methodology: The ALRM Framework
The authors propose ALRM (Agentic LLM for Robot Manipulation), a framework designed to bridge natural language reasoning with reliable robotic execution. The architecture is built around a ReAct-style reasoning loop and comprises three core modules:

1. Task Planner Agent: Generates iterative cycles of thought and action to decompose high-level user requests into subtasks. It dynamically creates subtasks based on environmental observations and feedback from the executor.
2. Task Executor Agent: Converts natural language subtasks into executable actions. This agent operates in two complementary modes:
   • Code-as-Policy (CaP): The LLM generates Python code that directly calls available control functions (e.g., pick, place, move) in a single run. This mode is faster and supports models without tool-calling capabilities but is brittle to code generation errors.
   • Tool-as-Policy (TaP): The LLM executes actions via nested tool calls in interactive steps. This mode allows for intermediate reasoning and error correction (e.g., referencing an incorrect object) but incurs higher latency and relies on the model's tool-calling reliability.
3. API Server: A RESTful server that interfaces with the robotic backend (Interbotix wx250s arm in Gazebo simulation via ROS and MoveIt), handling perception queries and motion commands.

Key Contributions

• ALRM Framework: A novel, modular agentic framework supporting both CaP and TaP execution modes, enabling iterative planning and reflection.
• New Benchmark: A simulation benchmark comprising 56 tasks across three environments (Kitchen utensils, Boxes, Fruits). The benchmark features three canonical tasks per environment, each paraphrased into five variations covering four linguistic categories: Lexical, Syntactical, Semantics, and High-Level Reasoning (HLR). This design specifically tests the agent's ability to handle diverse instructions and complex reasoning (e.g., selecting objects based on properties like "lowest calories" rather than explicit names).
• Systematic Evaluation: A comprehensive evaluation of ten LLMs (both closed-source and open-source) across both execution modes, measuring success rates and latency.
• Simulation Environment: A reusable Gazebo-based simulation environment integrated with ROS and MoveIt to support the deployment and testing of LLM-driven manipulation.

Experimental Results
The authors evaluated ten LLMs, including GPT-5, Claude-4.1-Opus, DeepSeek-V3.1, and open-source models like Falcon-H1-7B and Llama-3.1-8B.

• Model Performance:
  • Closed-Source: Claude-4.1-Opus emerged as the top performer, achieving average success rates of 93.5% in TaP mode and 92.6% in CaP mode.
  • Open-Source: Falcon-H1-7B was the top open-source model under CaP, achieving an 84.3% success rate with significantly lower latency (24.89s) compared to larger models.
• Mode Comparison: Large-scale LLMs generally achieved higher success rates with TaP (leveraging nested reasoning) but at the cost of increased latency. Conversely, small-scale models (up to 8B parameters) struggled with TaP due to difficulties in handling nested calls but performed competitively with CaP.
• Task Categories: Large-scale models performed consistently well across all linguistic categories, including high-level reasoning. Small-scale models excelled in syntactical variations but showed significant performance drops in high-level reasoning tasks, particularly when required to generate code (CaP) or handle complex tool chains (TaP).

Significance and Claims
The paper claims that ALRM provides a scalable, interpretable, and modular approach to robotic manipulation. The results demonstrate that LLMs can effectively solve high-level, abstract robotic tasks when equipped with agentic frameworks that support reflection and closed-loop revision. The authors highlight that while closed-source models currently lead in performance, specific open-source models (like Falcon-H1-7B) offer a viable, low-latency alternative for code-based execution. The work underscores the importance of evaluating agents not just on task completion, but on their robustness to linguistic diversity and their ability to reason over object properties.

Limitations and Future Directions
The authors acknowledge that the framework's performance is dependent on prompt engineering and that the current benchmark is limited to three environments and pick-and-place tasks. Future work is suggested to expand the benchmark to include navigation and more diverse task types, as well as to integrate real-world perception components to validate performance outside of simulation.