Technical Summary: CRAFT – Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks

1. Problem Formulation

The paper addresses the challenge of applying Multi-Agent Reinforcement Learning (MARL) to real-world multi-robot coordination tasks. While MARL offers a framework for decentralized decision-making, its application in robotics is hindered by:

• High-dimensional continuous joint action spaces.
• Sparse and complex reward design: Defining rewards that capture both individual behaviors and necessary team coordination is difficult.
• Non-stationarity: Agents learning concurrently create a shifting environment for one another.
• Long-horizon tasks: Tasks requiring contact-rich behaviors over multiple stages often fail because small errors in early interactions prevent later success.

The authors posit that humans learn complex coordination through "coaching"—a process involving curriculum design (breaking tasks into subtasks), providing learning signals (rewards), and evaluating progress. They argue that Foundation Models (Large Language Models - LLMs and Vision-Language Models - VLMs) can automate this coaching role, decomposing tasks, generating rewards, and refining them based on visual and quantitative feedback.

2. Methodology: The CRAFT Framework

CRAFT (Coaching Reinforcement learning Autonomously using Foundation models for learning coordination Tasks) operates as an automated coaching loop consisting of four key modules:

2.1 Curriculum Generation Module

A Curriculum LLM decomposes a long-horizon target task ($M_{target}$) into a sequence of $K$ subtasks ($M_1, \dots, M_K$).

• Process: The LLM receives a natural language description of the task and environment. To mitigate the stochasticity of LLM outputs, the system generates multiple candidate curricula and prompts the LLM to refine them into a single, coherent curriculum.
• Grounding: Subtasks are described using specific environment state variables (e.g., "minimize Euclidean distance to handle") rather than vague natural language to ensure precise reward design.

2.2 Reward Function Generation Module

For each subtask $M_k$, a Reward Generation LLM translates the natural language description into an executable Python reward function ($R_k$).

• Context: The LLM is provided with environment descriptions, helper functions (to process high-dimensional states), and reward functions from previous subtasks to prevent catastrophic forgetting.
• Structure: The reward is decomposed into interpretable components (e.g., lift_reward, balance_reward) to facilitate later analysis.
• Training: A decentralized policy $\pi_k$ is trained using a Centralized Training and Decentralized Execution (CTDE) algorithm (specifically MAPPO).

2.3 Policy Evaluation Module

Before advancing to the next subtask, an Evaluation VLM assesses the trained policy $\pi_k$.

• Inputs: The VLM analyzes rollout images (visual snapshots) and state-variable trajectories.
• Decision: It determines if the policy successfully completed the subtask.
  • Success: The curriculum proceeds to $M_{k+1}$.
  • Failure: The VLM provides a reason for failure, triggering the refinement loop.

2.4 Reward Refinement Module

If a policy fails, the system enters a refinement loop (up to $J$ iterations, typically $J=3$):

1. Advice Generation: An Advice VLM analyzes the failure reason, the current reward code, and reward-component learning curves (plots showing how specific reward terms evolved during training). It generates free-form advice on how to modify the reward (e.g., "increase the weight of the lift term," "remove the square on elevation").
2. Reward Refinement: A Refine LLM takes the failed reward code and the VLM's advice to generate a new, refined reward function ($R^{j+1}_k$).
3. Retraining: The policy is retrained with the new reward. If all attempts fail, the best policy from the iterations is selected, and the curriculum moves to the next subtask.

Warm-Starting: When moving to a new subtask, the policy weights are initialized from the successful policy of the previous subtask. To prevent "loss of plasticity" (convergence to local optima), the system resets the exploration parameters (policy standard deviation) and the value network weights while preserving the actor weights.

3. Key Contributions

1. CRAFT Framework: A novel framework that leverages the reasoning capabilities of foundation models to autonomously coach multi-robot teams through long-horizon coordination tasks.
2. Automated Curriculum and Reward Design: The integration of LLMs for task decomposition and reward generation, coupled with VLMs for visual evaluation and reward refinement, reducing the need for human-designed curricula and reward functions.
3. Empirical Validation: Demonstration of CRAFT's effectiveness on complex multi-robot tasks (multi-quadruped navigation and bimanual manipulation) where standard MARL and baseline LLM methods fail.
4. Real-World Transfer: Successful zero-shot transfer of learned policies from simulation to physical hardware (Unitree Go1 and Go2 quadrupeds) without fine-tuning.

4. Experimental Results

The authors evaluated CRAFT on three environments: Quadruped Gate (narrow gate passing), Quadruped Seesaw (balancing and climbing), and Two-Arm Lift (cooperative pot lifting).

• Performance vs. Baselines:
  • CRAFT achieved the highest success rates in Seesaw (59.33%) and Lift (94.67%), and remained competitive in Gate (78.33%).
  • Baselines: "Env Reward" (human-designed) and "Example Reward" achieved near-zero or very low success rates. "From Scratch" (CRAFT rewards without curriculum) performed significantly worse than CRAFT, highlighting the necessity of the curriculum.
  • LLM Baselines: Compared to Eureka (LLM reward search without curriculum) and CurricuLLM (LLM curriculum without VLM refinement), CRAFT outperformed both. Eureka often learned policies with unnatural joint configurations that were physically implausible, while CurricuLLM suffered from lower reliability due to the lack of reward refinement.
• Reward Refinement Efficacy: The VLM-guided refinement loop significantly improved the "Curriculum Success Ratio" (the fraction of trials resulting in a successful policy) and the "Maximum Success Rate" compared to CurricuLLM.
• VLM Reliability: The VLM policy evaluation module achieved 85.71% accuracy against ground truth, with errors being conservative false negatives (rejecting successful policies), which is tolerable as it simply triggers further refinement.
• Hardware Transfer: In the Quadruped Gate task, the policy trained in simulation achieved a 65% success rate on real robots over 20 runs without any fine-tuning, demonstrating robust sim-to-real transfer.

5. Significance and Claims

The paper claims that CRAFT demonstrates the viability of using foundation models as autonomous coaches for multi-robot systems. Its significance lies in:

• Reducing Human Intervention: Automating the tedious process of designing curricula and reward functions for complex coordination tasks.
• Enabling Complex Coordination: Successfully learning behaviors that are difficult to acquire via direct training or static reward design, particularly in long-horizon, contact-rich scenarios.
• Physical Plausibility: The staged curriculum approach guides agents toward physically feasible behaviors (e.g., stable grasps before lifting) rather than optimizing solely for terminal success, which often leads to unnatural motions in direct optimization methods like Eureka.

Limitations Acknowledged:

• Stochasticity: The reliance on foundation models introduces variability; some trials may fail due to poor curriculum or reward generation, though this is manageable in simulation by running multiple attempts.
• VLM Dependence: The evaluation module relies heavily on numerical state trajectories rather than purely visual information, potentially limiting the detection of visually grounded failures (e.g., poor spatial coordination).
• Scalability: The current approach requires multiple LLM/VLM queries per subtask, which may be computationally expensive, though the authors note it is sufficient for the tested tasks.

The authors conclude that while foundation models introduce stochasticity, their ability to decompose tasks and refine rewards autonomously offers a promising path toward scalable and reliable multi-robot coordination in real-world environments.