Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperation

1. Problem Statement

Decentralized multi-agent cooperation (involving humans and robots, or multiple robots) faces a fundamental challenge: agents often lack knowledge of each other's goals and plans, making collision avoidance and efficient collaboration difficult. While natural language communication can bridge this gap, grounding language instructions into safe, executable path plans remains a significant hurdle.

Existing approaches suffer from two main limitations:

• Safety: Directly generating paths or actions via Large Vision-Language Models (VLMs) often results in physically invalid trajectories (e.g., collisions with obstacles).
• Interpretability: "Black box" neural policies make it difficult for humans to understand why a robot chose a specific path or how it interpreted an instruction.

The authors propose CaPE (Code as Path Editor), a framework that treats language not as a direct generator of motion, but as a guide to edit existing, safe path plans generated by a classical planner.

2. Methodology: CaPE Framework

CaPE operates as a plug-and-play module that integrates a Vision-Language Model (VLM) with a model-based planner. The workflow consists of four key components:

A. Joint Planner

Before editing, the system generates a set of candidate paths.

• Target Agent: Uses a homotopy-aware path planner (based on RRT) to generate multiple collision-free candidate paths belonging to different homotopy classes (topologically distinct routes).
• Other Agents: Predicts the future paths of other agents (humans or robots) based on their goals (known or inferred).
• Output: A set of candidate trajectories for the target agent and predicted trajectories for others, providing the physical context for the VLM.

B. Path Editing Program Synthesis (VLM)

Instead of outputting raw coordinates, the VLM generates a program in a Domain-Specific Language (DSL) to edit the selected path.

• Input: Multimodal context including the map, candidate paths, predicted paths of others, and natural language instructions.
• DSL Operations: The VLM selects a base path and applies discrete operations such as:
  • select_path: Choosing a specific homotopy class.
  • modify_translation/rotation: Shifting waypoints locally.
  • wait: Pausing at a specific step.
  • insert_waypoint: Adding new points to the trajectory.
• Training: A small VLM (e.g., Qwen 2.5 7B) is fine-tuned on 50k synthetic examples to synthesize these programs efficiently, avoiding the latency of massive models.

C. Safety Verification

This is the core safety mechanism.

• Before execution, a Verifier (using the same model-based planner) checks every line of the generated code.
• It simulates the proposed edits against physical constraints and the predicted paths of other agents.
• Any operation that would cause a collision or violate constraints is rejected. The system outputs a verified, safe path.

D. Execution

The verified path is executed by the robot. If the environment changes or new instructions arrive, the cycle repeats.

3. Key Contributions

1. Novel Framework (CaPE): A method for language-guided program synthesis that edits motion plans rather than generating them from scratch. This ensures physical safety through planner-based verification.
2. Plug-and-Play Integration: The system is designed to be integrated into diverse robotic systems for multi-robot and human-robot cooperation without requiring retraining of the underlying robot controller.
3. Efficient VLM Training: A method for fine-tuning a small VLM to synthesize motion editing programs, balancing performance with low computational cost and latency.
4. New Benchmarks: Introduction of simulated and real-world benchmarks for multi-agent cooperation, including autonomous driving (parking lots), household object rearrangement, and joint carrying.
5. Real-World Validation: Demonstration of sim-to-real generalization in a real-world joint-carrying task with a Stretch 3 robot and human participants.

4. Experimental Results

The authors evaluated CaPE across three distinct scenarios, comparing it against baselines like "Planner-only," "VLM Agents" (direct action generation), and "VLM Waypoint Generators."

• Multi-Robot Coordination (SimWorld - Parking Lot):
  • 2-Robot Scenario: Baseline VLMs failed almost entirely (Success Rate < 20%). CaPE achieved a 76.6% success rate with large VLMs and 90.0% with the fine-tuned model.
  • 3-Robot Scenario: Baselines failed completely (0% success). CaPE maintained a 60.0% success rate, demonstrating scalability.
• Human-Robot Teaming (VirtualHome - Household):
  • In cluttered, multi-room environments, VLM agents failed (0% success).
  • CaPE with Gemini 3.0 Pro achieved an 80.0% success rate, significantly outperforming the Planner-only baseline (8.3%).
• Real-World Joint Carrying (Stretch 3 Robot):
  • Setup: 10 participants carried a PVC pipe with a robot through tight, obstacle-filled spaces.
  • Results: CaPE variants achieved a 70% success rate, compared to 20% for Planner-only and 10% for the best baseline (Waypoint generation).
  • Subjective Evaluation: Participants rated CaPE higher in safety, instruction comprehension, and overall collaboration ranking. The fine-tuned model was notably faster (2x inference speed) while maintaining high success.

5. Significance and Impact

• Safety First: By decoupling the "intent" (VLM) from the "execution" (Planner), CaPE guarantees that language instructions do not lead to unsafe physical actions. The verifier acts as a safety net.
• Interpretability: Because the VLM outputs a program using explicit DSL operations (e.g., "wait 2 steps," "move left 10cm"), humans can directly inspect the code to understand the robot's reasoning and coordination strategy.
• Generalization: The system successfully generalizes from synthetic training data to complex, unseen real-world environments and diverse human instructions.
• Paradigm Shift: CaPE proposes a shift from "Language-to-Action" to "Language-to-Plan-Edit," offering a more robust and transparent approach to interactive robot coordination.

In conclusion, CaPE demonstrates that combining the flexibility of VLMs with the rigor of classical planning creates a powerful, safe, and interpretable solution for complex multi-agent cooperation tasks.