Technical Summary: HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

Problem Statement

Cooperative Multi-Agent Reinforcement Learning (MARL) often operates under partial observability, where agents must coordinate based on limited local views. While communication is a standard mechanism to mitigate this, existing protocols typically treat messages as flat, dense vectors. These vectors are learned projections of an agent's entire observation, detached from the intrinsic structure of that observation.

Many cooperative environments (e.g., cyber defense, team sports, or micromanagement) possess a natural hierarchical observation structure (e.g., groups of entities, subnets containing hosts, or units of specific classes). Current methods fail to exploit this structure within the message body itself. Instead, they either transmit unstructured summaries or impose hierarchy on the communication graph or agent roles (e.g., feudal architectures) rather than on the data being exchanged. This oversight ignores a rich source of inductive bias, leading to inefficient communication where agents transmit redundant or unstructured information.

Methodology: HiComm

The authors propose HiComm, a plug-in communication module that grounds messages directly in the sender's hierarchical observation. Unlike traditional sender-driven protocols where an agent encodes its full state, HiComm is receiver-driven and treats communication as structured information retrieval.

Core Mechanism

The observation space for each agent $i$ is modeled as a three-axis tensor $z_i \in G \times E \times F$, where:

• $G$: Coarse group axis (e.g., subnets, unit classes).
• $E$: Fine entity axis (e.g., hosts within a subnet, specific units).
• $F$: Feature vector axis.

HiComm transforms communication from transmitting a vector to retrieving a specific address-value pair $((g, \ell), F)$ from the sender's observation tensor. The process occurs in three differentiable stages using Straight-Through Gumbel-Softmax (ST-GS) for discrete selection:

1. Phase 1: Query Generation (Receiver-side)
   The receiver summarizes its local observation context and observability mask into a query vector $q_i$. This query represents the receiver's information need.

2. Phase 2: Hierarchical Sender Selection (Receiver-side)
   The receiver descends the hierarchy to select a sender:

   • Stage 1 (Group Selection): The receiver selects a topic group $g^*$ by scoring the query $q_i$ against learnable group keys $\kappa_g$.
   • Stage 2 (Agent Selection): Conditional on $g^*$, the receiver selects a specific sender $j^*$ from the team (excluding self) based on a context-conditioned affinity function $\Phi(j, g^*; h_i)$.
   • Note: This two-stage descent reduces the discrete decision space from $|G| \cdot |E| \cdot (N-1)$ to $|G| + (N-1) + |E|$, localizing credit assignment.

3. Phase 3: Entity Retrieval (Sender-side)
   The receiver transmits $(q_i, g^*)$ to the selected sender $j^*$. The sender then:

   • Selects a specific entity $\ell^*$ within group $g^*$ on its own local observation $z_{j^*}$, masking unobserved entries.
   • Returns the verbatim feature vector $F = z_{j^*}[g^*, \ell^*, :]$ to the receiver.
   • Crucially, the message is the raw observation entry, not a learned compression.

Training and Integration

HiComm is designed as a lightweight, stateless transform that integrates into standard MARL pipelines (e.g., IPPO, MAPPO) without altering the underlying policy or critic architectures.

• Optimization: The communication parameters $\theta_{comm}$ are updated alongside the policy parameters $\theta_{pol}$ via a single backward pass.
• Loss Function: The objective combines the standard PPO surrogate loss with a scalar information-gain shaping prior. This prior encourages the selection of coordinates that provide high information gain relative to the receiver's current state, preventing sender collapse.
• Bandwidth: The protocol transmits only the address (group and entity indices) and the raw feature vector, significantly reducing bandwidth compared to transmitting full observation summaries.

Key Contributions

1. HiComm Protocol: A receiver-driven communication module that utilizes the sender's intrinsic group-entity hierarchy. It replaces unstructured vector transmission with structured retrieval of specific observation slices.
2. Differentiable Implementation: A lightweight, plug-in design using ST-GS for discrete selection and a shared projection head, allowing seamless attachment to standard MARL algorithms without modifying the core policy architecture.
3. Empirical Validation: Demonstration across three diverse benchmarks (CAGE Challenge 4, SMACv2, Google Research Football) showing that HiComm matches or outperforms representative learned communication baselines (CACOM, T2MAC) while drastically reducing communication volume.

Experimental Results

The authors evaluated HiComm on three benchmarks with varying observation structures and coordination demands, using both IPPO (decentralized critic) and MAPPO (centralized critic) backbones.

• Performance (RQ1): HiComm closes a substantial fraction of the performance gap between partial observability (no communication) and full observability (full state sharing). In 9 out of 10 algorithm-scenario cells, HiComm matched or outperformed the baselines. Notably, under MAPPO in the CAGE Challenge 4 environment, HiComm recovered ~85% of the gap to full observability, significantly outperforming CACOM and T2MAC.
• Bandwidth Efficiency (RQ2): HiComm achieves these results while reducing communication volume by 5× to 23× per receiver per episode compared to baselines, and over 1000× compared to full state sharing. This is achieved by transmitting only the relevant feature slice rather than a flattened projection of the entire observation.
• Generality (RQ3): The performance gains and bandwidth efficiency hold across different critic architectures (IPPO vs. MAPPO) and different methods of defining the hierarchy (native subnets, semantic unit classes, and semantic player roles).

Significance and Limitations

The paper claims that HiComm introduces a novel dimension to MARL communication: observation-space hierarchy. By allowing the receiver to actively query specific coordinates in the sender's observation structure, the protocol avoids the "lossy projection" inherent in flat vector methods. The results suggest that exploiting the natural structure of observations leads to more efficient and effective coordination.

Limitations:
The authors explicitly state that the work lacks a formal theoretical analysis of communication efficiency. The demonstrated bandwidth advantage is based on empirical comparison rather than an information-theoretic bound proving why hierarchical addressing is provably more efficient than flat encoding at matched task performance. Establishing such a theory is identified as a primary direction for future work.

Conclusion:
HiComm demonstrates that treating communication as structured retrieval from a hierarchical observation space is a viable and efficient alternative to traditional learned communication protocols, offering significant bandwidth savings without sacrificing (and often improving) cooperative performance.