Technical Summary: Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning

Problem Statement

The integration of service robots into domestic environments requires systems capable of interpreting imprecise, incomplete, or context-dependent natural language instructions. While traditional Task and Motion Planning (TAMP) systems rely on manually defined symbolic rules (e.g., PDDL), they struggle with the open-vocabulary nature of human environments. Conversely, while Large Language Models (LLMs) demonstrate emergent reasoning capabilities for task decomposition, they face significant limitations when applied directly to robotics:

1. Lack of Physical Grounding: Standard LLMs are not inherently grounded in physical laws and often generate "hallucinated" plans that are kinematically or geometrically infeasible.
2. Monolithic Prompting Issues: Current prompting strategies often force a single model instance to handle both abstract logic (e.g., "What is a fruit?") and precise spatial mathematics (e.g., "Where is the coordinate relative to the plate?"). This leads to "token exhaustion" or confused reasoning where the model prioritizes task logic over spatial accuracy.
3. Perception-Action Gap: There is a disconnect between high-level semantic planning and low-level geometric execution, particularly in unstructured environments where object geometry and workspace constraints vary.

Methodology

The authors propose a hierarchical, language-driven framework that decouples high-level task planning from low-level spatial reasoning using a dual-module prompting architecture. The system operates within a tabletop environment and integrates the following components:

1. Vision Module

The system utilizes a YOLOX-GDRNet pipeline for object perception:

• Detection: YOLOX detects objects in an input image, providing 2D bounding boxes and class labels.
• Pose Estimation: Regions of Interest (ROI) are processed by a GDR-Net neural network to predict 6D poses (3D translation vector and quaternion orientation) relative to the camera frame.
• Geometric Data: The system accesses pre-calculated 3D models and diameters for detected objects to inform spatial reasoning.

2. High-Level LLM Agent (Task Planner)

• Architecture: Implemented using Python LangChain with the Llama3.2:70B model.
• Prompting Strategy: The agent operates via a ReAct-style loop (Thought, Action, Observation). It maintains a memory of interaction history to support iterative reasoning.
• Tools: The agent has access to tools for retrieving object lists, picking objects, placing objects, and releasing them. It is responsible for the overall task logic and sequencing but delegates precise coordinate calculation.

3. Low-Level LLM Sub-Module (Placing Reasoner)

• Architecture: A separate, specialized sub-module also powered by Llama3.2:70B.
• Function: This module handles 3D spatial constraints and geometric feasibility. It is unaware of the high-level agent's internal logic, ensuring modularity.
• Prompting Strategy: It utilizes few-shot prompting with a structured input consisting of object names, poses, and geometric properties (e.g., diameters).
• Objective: To map spatial language (e.g., "to the left of") to consistent geometric transformations and determine specific 3D coordinates for object placement, thereby preventing the high-level planner from becoming overwhelmed by low-level coordinate math.

Key Contributions

The paper outlines three primary contributions:

1. Dual-Module Architecture: A novel framework that decouples high-level task planning from low-level spatial reasoning to improve the reliability of generated robotic plans.
2. Specialized Spatial Sub-Module: A dedicated component designed to handle 3D spatial constraints and geometric feasibility, directly addressing the lack of physical grounding in standard LLMs.
3. Empirical Validation: Demonstration of the framework's efficacy in 24 real-world and simulated scenarios, showing significant improvements in executing complex human commands in unstructured environments.

Evaluation and Results

The system was evaluated across 24 test scenarios, including simple spatial commands, high-level instructions, and infeasible requests.

Quantitative Results

• Overall Success Rate: The system achieved an 86% overall task success rate.
• Scenario Breakdown:
  • Simple Commands (16 scenarios): 87.5% success rate, averaging 4 steps per task.
  • High-Level Commands (5 scenarios): 80.0% success rate, averaging 6 steps per task.
  • Infeasible Scenarios (3 scenarios): 100% success rate in correctly identifying impossibility, averaging 5 steps to determine infeasibility.
• Planning Duration: The average planning duration was 5 steps per task.

Qualitative and Spatial Analysis

• Human Evaluation: Three independent evaluators rated the alignment between initial commands and final scenes on a scale of 1–10. The mean rating was 6.86, with higher scores for tasks with clear spatial relationships (e.g., "Make a bowl of fruits" scored 9.67) and lower scores for ambiguous tasks (e.g., "Make a salty snack" scored 3.00).
• Spatial Reasoning Precision: The placing reasoner demonstrated the highest precision for "on top" and "next to" relations (lowest coordinate variance). Performance was less consistent for depth-based relations like "in front" and "behind," which showed higher variance.
• Physical Robot Trials: Experiments on a PAL Robotics Tiago manipulator confirmed that the system could reliably identify target objects and infer user intent in cluttered environments. Failures observed were confined to low-level execution (motion planning and calibration errors) rather than semantic or task-level reasoning.

Significance and Claims

The authors claim that their hierarchical approach effectively bridges the gap between natural language and 3D geometric relations, exhibiting a robust "spatial grammar" suitable for zero-shot placement. The paper asserts that:

• Modularity is Key: Isolating spatial geometry reasoning into a dedicated sub-module prevents the high-level planner from failing due to the complexity of coordinate math.
• Feasibility over Expressivity: The framework prioritizes execution reliability on real robots, trading broad action expressivity for physical feasibility.
• Future Direction: While the system shows promise, the authors note that real-world utility remains constrained by physical feasibility and environmental generalization. They suggest that future work should integrate geometric priors and physical simulators to refine predictions within dynamic, embodied settings.

The paper concludes that merging the symbolic flexibility of LLMs with grounded spatial perception offers a promising path toward intuitive and generalizable human-robot collaboration.