Technical Summary: Closed-Loop Vision-Language Planning for Multi-Agent Coordination (COMPASS)

Problem Statement

Cooperative Multi-Agent Reinforcement Learning (MARL) faces significant challenges in sample efficiency, interpretability, and generalization, particularly in complex, partially observable environments. While Large Language Models (LLMs) have demonstrated powerful planning capabilities, their application to multi-agent systems has been limited by two primary factors:

1. Text-Only Inputs: Existing LLM-based agents often rely solely on textual observations, failing to leverage multi-modal (visual) information crucial for real-time tactical decision-making.
2. Open-Loop Limitations: Many approaches utilize open-loop planning that does not adequately handle the non-Markovian, partially observable nature of multi-agent tasks, where agents must adapt dynamically to stochastic environments and incomplete information.

Furthermore, traditional MARL methods often struggle with scalability when central controllers are infeasible due to communication overhead, while decentralized approaches suffer from the discrepancy between local observations and global state.

Methodology: The COMPASS Framework

The authors introduce COMPASS, a decentralized, closed-loop framework that integrates Vision-Language Models (VLMs) to enable robust multi-agent coordination. The architecture consists of three synergistic components:

1. VLM-Based Closed-Loop Planner

Inspired by cognitive architectures, the planner employs four specialized VLM modules to iteratively perceive, reason, reflect, and act:

• Perception: Processes multi-modal inputs (visual screenshots and textual descriptions) to build a comprehensive environmental understanding. It operates at two levels: direct observation processing and collaborative information synthesis via a communication protocol.
• Task Reasoning: Decomposes high-level objectives (e.g., "defeat all enemy units") into concrete, executable sub-tasks based on current environmental feedback and team objectives.
• Self-Reflection: Continuously evaluates the quality of executed skills and task completion, generating verbal feedback to refine future decision-making.
• Actor: Selects and executes the most appropriate skills from a dynamic skill library based on the proposed sub-tasks and reflection feedback.

2. Adaptive Skill Synthesis

COMPASS utilizes a dynamic skill library containing executable Python code that represents tactical behaviors. This library evolves through two mechanisms:

• Demonstration Bootstrapping: To avoid inefficient exploration from scratch, the system initializes the skill library by analyzing expert demonstration videos (collected via MAPPO). VLMs extract strategic patterns and translate them into documented Python functions.
• Incremental Synthesis: During task execution, if existing skills are insufficient for a proposed sub-task, the VLM generates new Python scripts. These scripts follow a structured interface with score_target(unit) for dynamic prioritization and control_logic() for behavioral coordination. The system continuously refines these skills based on self-reflection and execution outcomes.

3. Structured Communication Protocol

To address partial observability without inducing hallucinations common in unconstrained natural language communication, COMPASS implements a structured, entity-based protocol:

• Multi-Hop Propagation: Agents share local observations of entities (allies and enemies) through a global memory. Information propagates through intermediate allies (multi-hop), allowing agents to infer the status and location of entities outside their direct field of view.
• Global Memory: A shared repository accessible to all agents, enabling the construction of a coherent, holistic understanding of the environment despite decentralized execution.

Key Contributions

1. Decentralized Closed-Loop Planning: Unlike prior LLM-based multi-agent works that often rely on centralized planning or open-loop execution, COMPASS implements a fully decentralized, closed-loop system where VLMs continuously adapt strategies based on real-time visual and textual feedback.
2. Code-as-Policy with Dynamic Libraries: The framework moves beyond fixed primitive actions by generating and refining executable Python code (skills) on the fly. This approach enhances interpretability and allows for the accumulation of reusable tactical knowledge.
3. Structured Entity-Based Communication: The paper proposes a communication protocol that mitigates the risks of meaningless chatter and hallucinations by focusing on structured, multi-hop propagation of entity information, effectively addressing the partial observability challenge.
4. Bootstrapping from Expert Data: The method demonstrates how to initialize a complex skill library from offline expert demonstrations, significantly reducing the sample complexity required to discover viable strategies.

Experimental Results

The framework was evaluated on the SMACv2 (StarCraft Multi-Agent Challenge v2) benchmark, which features increased complexity and partial observability compared to the original SMAC.

• Performance: COMPASS significantly outperformed state-of-the-art MARL baselines (including QMIX, MAPPO, HAPPO, HASAC, and MAT) in symmetric Protoss 5v5 scenarios, achieving a 57% win rate compared to QMIX's 27% (a 30 percentage point advantage).
• Robustness: The system maintained moderate performance in Terran scenarios and effectively handled asymmetric (5v6) settings, demonstrating the utility of the skill library and communication protocol in compensating for numerical disadvantages.
• Limitations: Performance was limited in Zerg scenarios (16% win rate), particularly against swarm units like Banelings and Zerglings. The authors attribute this to the mismatch between the VLM's query frequency (every 10–20 timesteps) and the sub-second tactical windows required for melee swarm coordination.
• Ablation Studies:
  • Communication: Removing multi-hop communication caused a drastic drop in win rate (from 57% to 6%), highlighting its critical role in partial observability.
  • Visual Input: Omitting image inputs resulted in a 10% performance drop, confirming the necessity of visual grounding for spatial understanding.
  • Self-Reflection: Removing the self-reflection module led to a 10% performance decrease.
  • Bootstrapping: Using only the initialized skill library (without incremental synthesis) yielded non-trivial performance but was significantly lower than the full COMPASS system, validating the need for dynamic skill evolution.

Significance and Claims

The paper claims that COMPASS demonstrates the viability of using Vision-Language Models to generate interpretable, tactical behaviors for cooperative multi-agent control under partial observability. It establishes that VLMs can effectively synthesize and refine code-based strategies when integrated with structured communication and closed-loop feedback.

However, the authors are modest regarding the scope of these findings. They explicitly acknowledge that the current approach is unsuitable for domains requiring rapid, continuous reactive control (such as Zerg swarm mechanics) due to the inference latency of VLMs. The paper posits that VLM-based planning is most advantageous in scenarios where strategic coordination dominates and actions remain valid across multiple timesteps. Future work is suggested to focus on reducing inference latency or developing hybrid architectures that delegate high-frequency decisions to learned reactive policies while reserving VLMs for strategic supervision.