🤖 AI

Modular Reinforcement Learning For Cooperative Swarms

This paper proposes a modular reinforcement learning approach that decomposes spatial interaction states into separate learning procedures to overcome the memory limitations of computationally constrained robot swarms, demonstrating its effectiveness in cooperative foraging tasks.

Original authors: Erel Shtossel, Gal A. Kaminka

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Erel Shtossel, Gal A. Kaminka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive group of tiny, very simple robots working together like a school of fish or a colony of ants. Their goal is to find scattered objects (like food) and bring them back to a central home base. This is called "swarm foraging."

The problem is that these robots are incredibly limited. They have very little memory (less than a tiny smartphone app) and very little brainpower. They can only see a few inches around them and can't talk to the whole group at once.

The Big Problem: The "State Explosion"

To learn how to work together, these robots need to use a type of learning called Reinforcement Learning. Think of this as a robot trying different moves and remembering which ones work best.

However, there's a catch. If a robot tries to remember every possible situation it could face, the number of situations grows so fast it becomes impossible to store.

The Analogy: Imagine you are trying to memorize a map of a city. If you try to remember every single street corner, every traffic light, and every pedestrian's position all at once, your brain would explode. In the robot world, this is called the "State Explosion." A standard robot trying to remember all these combinations would need a hard drive the size of a building, but it only has a memory chip the size of a grain of rice.

The Solution: The "Modular" Approach

The authors of this paper propose a clever workaround. Instead of one giant brain trying to remember everything, they give the robot eight tiny, specialized brains (one for each of its eight sensors).

The Analogy: Imagine a team of eight people trying to navigate a crowded room.
- The Old Way (Full State): One person tries to memorize the location of everyone in the room simultaneously. They get overwhelmed and forget things.
- The New Way (Modular): Each person only watches one specific direction. Person #1 only watches the front. Person #2 only watches the left. They don't worry about the whole room; they just worry about their specific slice of it.

Each of these "mini-brains" learns a simple rule: "If I see a robot in my direction, move away. If it's clear, keep going."

The "Council"

Once all eight mini-brains have made their suggestions, they need to decide on a single move for the robot. The paper calls this decision-making group "The Council."

The Analogy: The Council is like a committee meeting.
- The "Front" brain says, "Go forward!"
- The "Left" brain says, "Move right to avoid that guy!"
- The "Right" brain says, "Move left!"
- The Council takes all these conflicting votes, mixes them together using a mathematical formula (like averaging the opinions), and picks the best compromise direction.

What They Found

The researchers tested this in a computer simulation with up to 36 robots in different room layouts.

It Works: The modular approach (the eight mini-brains) performed just as well as, or sometimes better than, complex methods that tried to remember everything at once.
It's Efficient: It used a tiny fraction of the memory. Instead of needing to remember millions of scenarios, the robots only needed to remember a few dozen simple rules. This fits perfectly on their tiny microchips.
It's Robust: Even when the researchers changed the "reward system" (the way the robots were told they did a good job), the modular approach kept working, while the complex methods often crashed or failed.
Simple Moves are Better: They found that telling the robots to move in simple directions (vectors) worked better than trying to teach them complex, pre-programmed avoidance maneuvers.

The Bottom Line

This paper shows that you don't need a supercomputer to make a swarm of robots work together. By breaking a giant, impossible problem into many tiny, simple problems and letting a "council" vote on the final answer, you can create a swarm that is smart, cooperative, and fits on a very small, cheap robot.

Technical Summary: Modular Reinforcement Learning For Cooperative Swarms

Problem Statement

Cooperative robot swarms consist of collectives of computationally limited robots working toward a common goal with only local interactions. A primary challenge in applying Multi-Agent Reinforcement Learning (MARL) to such systems is the state explosion problem. To learn effective collision avoidance and navigation, robots must represent spatial interaction states. A naive, fully factored representation of these states (e.g., the presence or absence of neighbors in multiple sensor directions) grows combinatorially. For a robot with $k$ sensors, a binary state representation results in $2^k$ states.

This exponential growth is infeasible for physical swarm robots, which often possess limited memory (e.g., under 512KB RAM) and processing power (often lacking hardware floating-point support). Consequently, standard MARL approaches requiring deep neural networks or large state tables are impractical. Furthermore, robots operate as independent learners without access to global swarm states or the actions of peers, making reward alignment difficult.

Methodology

The paper proposes a modular (decomposed) state representation to address the memory and computational constraints of swarm robots. The core approach involves:

Decomposition of State: Instead of a single learner processing a complex joint state, the spatial state is decomposed by sensor direction. Each of the $k$ sensors (or spatial features) is assigned a separate, independent learning process.
- State Space Reduction: For $k$ features, the total number of states to store becomes linear ( $O(k)$ ) rather than exponential ( $O(2^k)$ ). In the experiments, 8 sensors with binary states result in $8 \times 2 = 16$ total states across all learners, compared to $2^8 = 256$ states for a full-state learner.
- Learning Process: Each learner is a multi-armed bandit algorithm (specifically Continuous UCB-1) that maps the local sensor reading (e.g., "collision" or "clear") to a recommended action.
Action Space and Fusion (The Council):
- Vectorial Actions: To enable the fusion of recommendations, the action space is defined as composable direction vectors (e.g., 8 discrete headings) rather than atomic collision-avoidance algorithms. This allows the outputs of different learners to be mathematically combined.
- Aggregation Mechanism: A fixed procedure, termed the "Council," aggregates the preferences of the $k$ independent learners. It assigns high preference values to the directions recommended by learners and lower values to neighboring directions based on a Gaussian distribution. The final action is selected by sampling from this accumulated probability distribution.
- Credit Assignment: There is no credit assignment between learners. All learners receive the same reward signal (the difference reward) based on the outcome of the fused action.
Reward Alignment: The system utilizes difference rewards ( $\Delta$ ), which approximate the marginal contribution of a robot to the collective utility. This allows independent learners to align their self-interested goals with the swarm's collective goal without global communication.

Key Contributions

Modular Representation for Spatial States: The paper introduces a decomposed state representation where each spatial feature (sensor direction) is handled by a separate learner. This reduces the memory footprint from exponential to linear relative to the number of sensors.
Composable Action Space: The authors demonstrate the necessity and efficacy of using vectorial action spaces that can be fused, contrasting them with the atomic algorithmic actions used in prior swarm learning work, which cannot be arbitrarily merged.
Robustness to Reward Variations: The modular approach is shown to be robust to changes in reward functions (e.g., switching from difference rewards to self-interested rewards), whereas full-state learners often fail under such conditions.

Experimental Results

The approach was evaluated in high-fidelity simulations (ARGoS3) using Krembot robots in a cooperative foraging task across three arena configurations (varying base locations and densities).

Performance vs. Full-State Learners:
- In low-density scenarios, the modular approach performed comparably to fixed navigation algorithms (Dynamic Window) and full-state learners (R-Learner, Q-Learning).
- In high-density scenarios, full-state learners (specifically R-Learner) eventually outperformed the modular approach, likely due to the richer state information available to them. However, the modular approach maintained stable performance without the memory overhead.
- In complex environments (e.g., corner bases causing high collision rates), the modular approach matched the performance of Dynamic Window but struggled to keep pace with full-state learners as density increased.
Robustness: When the reward function was changed from the aligned difference reward ( $\Delta$ ) to a raw self-interested reward ( $\Omega$ ), the modular representation maintained its performance, while the stateful R-Learner showed a dramatic decline.
Action Space Comparison: Experiments comparing learning with vectorial actions versus learning with atomic collision-avoidance algorithms (Repel, Dynamic Window, Aggression) showed that learning to use vectorial actions yielded improved results.

Significance and Claims

The paper claims that the modular representation offers a practical solution for deploying MARL on resource-constrained swarm robots. While it does not always outperform full-state learning algorithms in terms of raw performance in high-density scenarios, it provides dramatic memory savings and robustness where full-state methods fail or are infeasible.

The authors emphasize that this approach is particularly well-suited for spatial states because it mirrors the structure of potential fields and naturally admits a composable action space. The work suggests that for physical swarms with limited RAM (often <200KB), the trade-off of slightly lower optimality in dense scenarios for the ability to learn at all is significant. The paper concludes by noting that the modular representation allows for learning in settings where traditional state-explosion problems would otherwise render reinforcement learning irrelevant.