Technical Summary: GuideFetch

Problem Statement

The paper addresses the challenge of coordinating heterogeneous multi-robot teams for assistive tasks that require both navigation and object retrieval. Specifically, it targets scenarios where a blind user requires a guide dog to navigate to a destination (e.g., a seat) while a second assistive robot concurrently retrieves an object (e.g., a cup of coffee) and delivers it to the same location.

Current approaches often develop navigation (wayfinding, safety) and manipulation (grasping, delivery) as separate systems. While Large Language Models (LLMs) have been used for task decomposition and capability-aware allocation, a critical gap remains: a syntactically valid LLM plan does not guarantee that the assigned robots possess the necessary capabilities, nor does a smooth controller trajectory guarantee successful task completion (e.g., a grasp may fail, or an object may be dropped). The paper posits that for heterogeneous teams, coordination must go beyond executing independent skill sequences; it requires verifying that symbolic progress corresponds to verified physical completion, managing temporal dependencies, and distinguishing between plan validity and mission success.

Methodology

The authors introduce GuideFetch, a coordination framework that decouples task specification from robot execution and physical verification. The system operates within the InternUtopia simulation environment (built on GRUtopia), utilizing a Unitree Go2 as the "guider" and a Unitree B2 with a Z1 arm as the "fetcher."

Core Architecture

1. LLM Planning & Schema Instantiation:

   • An LLM (specifically a locally served Qwen3.5 model) receives a natural language instruction and a selected schedule (sequential or parallel).
   • The LLM instantiates a schedule-conditioned four-action schema. The actions are drawn from a fixed vocabulary: navigate, move, grasp, and carry.
   • The output is structured as a JSON object containing step indices, robot assignees, skills, targets, and parameters.

2. Deterministic Validation & Normalization:

   • Before execution, a deterministic validator normalizes robot, skill, and target aliases against a registered scene registry ($\Omega$).
   • The validator checks that:
     • All four expected actions are present.
     • Assignees possess the requested skills (e.g., the guider cannot grasp).
     • Targets are registered in the scene.
     • The action structure adheres to the selected schedule (e.g., parallel plans must start with simultaneous navigation and motion).
   • Invalid assignments are rejected immediately; no fallback or replay is performed during the validation phase.

3. Compilation & Execution:

   • Validated records are compiled into a task graph ($G=(V, E)$) defining nominal dependency levels.
   • Sequential Schedule: The fetcher performs the entire mission: visits the destination (empty), returns to the pickup, grasps, lifts, and delivers.
   • Parallel Schedule: The guider navigates to the destination (empty) while the fetcher simultaneously moves to the pickup, grasps, lifts, and carries.
   • Execution runs under a shared simulation clock. Branches progress based on local state events. The guider does not wait for the fetcher, and the fetcher does not wait for the guider, provided dependencies within their own branches are met.

4. State-Based Verification:

   • Mission success is not assumed from plan validity or trajectory smoothness. It is determined by Boolean variables computed from controller/simulator states:
     • $b_{visit}$: Arrival at the final station.
     • $b_{pickup}$: Arrival at the pickup station.
     • $b_{grasp}$: Bilateral contact with the object.
     • $b_{lift}$: Object rise of at least 30 mm.
     • $b_{retain}$: Retention of the object (20 mm rise, contact in 80% of frames, tilt < 17°) for 1.0s.
     • $b_{delivery}$: Arrival at the final station with the object.
   • Operational success ($y_{op}$) is the logical AND of all these conditions.

Key Contributions

• GuideFetch Framework: A system that integrates LLM planning with deterministic validation and state-based physical verification for heterogeneous robot teams.
• Schedule-Conditioned Schema: A method where the LLM generates plans constrained by a specific execution schedule (sequential vs. parallel), ensuring the output is executable by the specific robot capabilities.
• Decoupling of Validity and Completion: The framework explicitly separates the validity of the symbolic plan from the verified completion of the physical mission, using terminal state checks to gate success.
• Controlled Comparative Study: A rigorous evaluation across 90 matched scene/seed combinations (360 total executions) comparing sequential and parallel schedules using both scripted and online LLM planners.

Experimental Results

The study was conducted in three environments (Home, Supermarket, Hospital) with 30 seeds per scene.

• Plan Validity: All 180 online LLM responses passed validation on the first attempt without fallback or replay. Every online plan matched its corresponding scripted plan, indicating the LLM consistently produced the required schema.
• Operational Success:
  • Sequential execution: 72/90 successes (80.0%).
  • Parallel execution: 71/90 successes (78.9%).
  • The difference in success rates was attributed entirely to fetcher manipulation failures (grasp, lift, or retention), not navigation or planning errors. Every parallel guider reached its goal.
• Makespan (Time Efficiency):
  • In the 56 cases where both schedules succeeded, parallel execution reduced the mean makespan from 164.78s to 96.79s, a 41.3% reduction (1.70× speedup).
  • The speedup varied by scene: 39.5% in Home, 19.1% in Supermarket, and 48.6% in Hospital.
  • In Home and Supermarket, the speedup came from temporal overlap (concurrent motion). In Hospital, it also resulted from removing the fetcher's long unloaded return trip.
• Failure Analysis: All operational failures occurred during pre-transport manipulation (grasp, lift, retention). No failures occurred in plan validation, pickup arrival, or guider navigation.

Significance and Claims

The paper claims that GuideFetch successfully demonstrates that heterogeneous role specialization and action overlap can significantly shorten assistive missions without altering terminal objectives.

Key claims include:

1. Validity $\neq$ Success: A syntactically valid LLM plan or a smooth controller trajectory is insufficient to guarantee task completion; physical state verification is required.
2. Concurrency Benefits: Parallel execution of navigation and retrieval is feasible and efficient, provided that role assignment respects robot capabilities and state checks govern progress.
3. Robustness of LLMs: Within the tested schema and templates, the LLM consistently generated valid, executable plans that matched scripted baselines.
4. Bottleneck Identification: In the current setup, the limiting factor for mission completion is manipulation stability (grasp and retention), not navigation or planning.

The authors maintain a modest tone, noting that the language interface is limited to tested templates and a fixed action inventory, and that the simulation environment uses registered routes with disabled background physics. They do not claim the system is ready for unstructured real-world deployment but rather that it provides a controlled framework for studying the intersection of LLM planning, heterogeneous coordination, and physical verification.