M$^{2}$GRPO: Mamba-based Multi-Agent Group Relative Policy Optimization for Biomimetic Underwater Robots Pursuit

1. Problem Statement

The paper addresses the challenges of cooperative pursuit-evasion (PE) tasks involving biomimetic underwater robots (specifically robotic sharks). The core difficulties in this domain include:

• Long-Horizon Decision Making: Pursuit tasks require planning over extended timeframes, which is difficult for standard models that struggle with long-term dependencies.
• Partial Observability: Robots operate with limited local sensing, requiring them to rely on historical observation sequences to infer the state of the environment and the evader.
• Inter-Agent Coordination: Effective capture requires dynamic coordination among multiple pursuers with evolving interaction topologies.
• Resource Constraints: Underwater robots have limited computational power, necessitating algorithms that are both expressive and computationally efficient.

Existing methods often rely on Multi-Layer Perceptrons (MLPs) which fail to capture long-term temporal dependencies, or Transformers which incur high computational costs ($O(N^2)$ complexity) unsuitable for onboard deployment.

2. Methodology: M2GRPO

The authors propose M2GRPO, a framework integrating a Mamba-based policy network with Multi-Agent Group Relative Policy Optimization (MAGRPO) under the Centralized Training with Decentralized Execution (CTDE) paradigm.

A. Mamba-Based Policy Network

Instead of standard MLPs or Transformers, the policy network utilizes Mamba (Selective State Space Models) to handle temporal and relational data efficiently.

• Temporal Modeling: Uses a Mamba2 module to process historical observation sequences ($h_t$). It leverages input-dependent gating and segmented recursion to capture long-range temporal dependencies with linear time complexity, replacing the quadratic complexity of Transformers.
• Relational Modeling: Uses a BiMamba2 (Bidirectional Mamba) module to process current local observations (self-state, evader info, and collective features of neighbors) to extract interaction features.
• Fusion: A Multi-Head Attention (MHA) mechanism fuses the temporal features (Key/Value) and relational features (Query) to create a unified latent representation.
• Action Output: The fused representation is mapped to the parameters of a Gaussian distribution ($\mu, \sigma$) to sample continuous, bounded actions.

B. Multi-Agent Group Relative Policy Optimization (MAGRPO)

To improve training stability and reduce computational cost, the authors extend Group Relative Policy Optimization (GRPO) to multi-agent settings.

• Elimination of Value Networks: Unlike traditional PPO or MAPPO, MAGRPO does not require a separate critic/value network.
• Group-Normalized Advantages: For each agent, advantages are calculated by normalizing the episodic returns of parallel environments within a group.
  $$A^i_j = \frac{R^i_j - \bar{R}^i}{\text{Std}(R) + \tau}$$
  Where $R^i_j$ is the return in the $j$-th environment, and $\bar{R}^i$ is the group mean.
• PPO-Clip Objective: The policy is updated using a clipped objective function based on these relative advantages, ensuring stable updates without the instability often associated with value function approximation in multi-agent systems.

C. Reward Design

The reward function combines three components to guide learning:

1. Capture Reward ($r_{cap}$): Fixed positive reward upon successful capture.
2. Auxiliary Reward ($r_{aux}$): Encourages minimizing the distance to the evader.
3. Safety Reward ($r_{safe}$): Penalizes agents approaching the pool boundaries to prevent unsafe behaviors.

3. Key Contributions

1. Novel Architecture: Proposes the first integration of Mamba (Selective SSM) into a multi-agent policy for underwater robotics, effectively modeling long-horizon temporal dependencies and inter-agent relationships with linear computational complexity.
2. Algorithmic Innovation: Develops MAGRPO, a multi-agent extension of GRPO that removes the need for explicit value networks, significantly reducing memory usage and training complexity while maintaining stability.
3. Real-World Validation: Successfully deploys the learned policy on physical biomimetic robotic shark prototypes in a real-world pool environment, demonstrating the feasibility of the approach beyond simulation.

4. Experimental Results

The framework was evaluated through extensive simulations and real-world experiments against strong baselines (MAPPO, HAPPO, MASAC).

• Effectiveness (Success Rate):
  • M2GRPO achieved a 97% success rate against a learned evader and 93% against a random evader.
  • It outperformed all baselines (MAPPO, HAPPO, MASAC) by a significant margin, attributed to its superior ability to model long-term intent and coordinate agents.
• Efficiency (Capture Steps):
  • M2GRPO required the fewest average steps to capture the evader, demonstrating more forward-looking planning compared to baselines that often made redundant moves.
• Scalability:
  • As the number of pursuers increased (from 2 to 6), M2GRPO maintained the highest success rate.
  • Baselines like MASAC and MHPPO showed performance degradation or linear increases in training overhead as agent count grew, whereas M2GRPO's attention-based fusion scaled efficiently.
• Ablation Studies:
  • Removing the interaction encoding caused the largest performance drop (from 92% to 81%), highlighting the critical role of relational modeling.
  • Replacing Mamba with a standard MLP resulted in a severe drop (to 66%), confirming the necessity of the SSM backbone for temporal reasoning.
• Real-World Experiments:
  • Physical tests with robotic sharks confirmed the algorithm's ability to adapt to dynamic water conditions, successfully executing complex strategies like "cornering" and "encirclement" to capture a mobile evader.

5. Significance

This work represents a significant advancement in underwater multi-robot cooperation. By combining the linear-time efficiency of Mamba with the sample efficiency of Group Relative Policy Optimization, M2GRPO solves the trade-off between expressiveness and computational cost.

• Practical Application: It provides a scalable solution for real-world tasks such as underwater inspection, search-and-rescue, and environmental monitoring where robots must operate with limited onboard compute and partial observability.
• Theoretical Impact: It demonstrates that state-space models (SSMs) can effectively replace Transformers in multi-agent reinforcement learning, offering a new direction for resource-constrained robotic systems.