🔬 condensed matter

Learning to flock in open space by avoiding collisions and staying together

This paper demonstrates that cohesive, high-order flocking in open space can emerge through multi-agent reinforcement learning, where agents optimize a local cost function to balance collision avoidance and group cohesion, resulting in dynamics that closely mirror the structural and behavioral characteristics of real starling flocks.

Original authors: Martino Brambati, Antonio Celani, Marco Gherardi, Francesco Ginelli

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Martino Brambati, Antonio Celani, Marco Gherardi, Francesco Ginelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, chaotic dance floor where thousands of dancers (the "agents") are trying to move together without bumping into each other or getting lost in the crowd. For a long time, scientists have wondered: How do birds in a flock, like starlings, manage to fly in such a perfect, synchronized wave without crashing?

This paper uses a computer simulation to teach a group of digital "dancers" how to do exactly that, using a method called Reinforcement Learning. Think of this like training a dog: you give them a treat when they do something right and a gentle "no" when they do something wrong. Eventually, they figure out the best way to behave.

Here is the story of what the researchers discovered, broken down into simple concepts:

1. The Two Rules of the Dance Floor

The researchers gave their digital dancers two main goals to balance:

Stay Together: Don't drift too far away from your neighbors, or the group falls apart.
Don't Crash: Don't get too close, or you'll collide.

They set up a "scorecard" (a cost function) for the dancers. If a dancer got too far from the group, the score went down. If they got too close and risked a crash, the score went down even harder. The dancers had to learn how to move to keep their score as high as possible.

2. The "Aha!" Moment: Avoiding Crashes is Key

The most surprising discovery was that avoiding collisions is actually the secret sauce for flocking.

The "Swarm" Mistake: When the researchers told the dancers only to stay together (ignoring the risk of crashing), the group turned into a messy, disordered blob. They huddled tightly together but moved in random directions, like a swarm of bees or a crowd of people rushing a door. This is called swarming.
The "Flock" Success: When the researchers added a penalty for getting too close (simulating the fear of a crash), the behavior changed completely. The dancers learned to space themselves out just enough to avoid hitting each other. This forced them to align their movements—facing the same direction and moving in a smooth, coordinated wave. This is flocking.

The Analogy: Imagine driving on a highway. If you only care about staying with your carpool group, you might drive erratically to keep up. But if you are also terrified of hitting the car in front of you, you naturally fall into a smooth, synchronized lane, matching the speed and direction of everyone else. The fear of the crash creates the order.

3. How the Dancers Learned to Move

The digital dancers learned a simple two-step strategy based on how far away their neighbors were:

When neighbors are far away: They used a mix of "pulling" (attraction) to get closer and "copying" (alignment) to face the same way.
When neighbors are close: They stopped pulling and focused almost entirely on copying. They aligned their direction perfectly with their neighbors. This strong alignment at close range is what keeps the flock tight and moving as one unit.

4. Centralized vs. Decentralized Learning

The researchers tested two ways of teaching the dancers:

Centralized (The Hive Mind): All dancers shared their learning. If one dancer figured out a good move, everyone learned it instantly. This worked very well.
Decentralized (The Solo Learner): Each dancer learned on its own, without talking to the others. This also worked, but it took longer and was a bit harder for very large groups. However, for medium-sized groups (like a typical starling murmuration), both methods led to the same beautiful, synchronized result.

5. The Result: A Liquid-Like Flock

The final result looked just like real bird flocks. The group moved as a single, cohesive unit with high order (everyone facing the same way). Interestingly, the dancers didn't hold fixed positions like soldiers; they flowed past each other like a liquid, constantly swapping neighbors while maintaining the overall shape of the flock. This matches what scientists observe in real starling flocks in the sky.

The Bottom Line

The paper suggests that the reason some animals (like starlings) fly in organized flocks while others (like midges) just swarm in a messy ball might be simple: Flocking is the smartest way to stay together without crashing into each other. The need to avoid a collision forces the group to align, creating that breathtaking, synchronized dance we see in nature.

Technical Summary: Learning to Flock in Open Space by Avoiding Collisions and Staying Together

Problem Statement
The paper addresses a fundamental challenge in the modeling of collective motion: how to maintain group cohesion in open, boundless space. While minimal models like the Vicsek model successfully generate long-range order in periodic systems through alignment interactions, they fail in open systems. Purely alignment-based interactions lead to diffusive spreading and group disintegration, while purely attractive interactions often result in disordered swarming or milling rather than cohesive flocking. The authors investigate how self-propelled particles (SPPs) can learn to balance alignment and attraction to achieve cohesive, highly ordered flocking without relying on pre-defined interaction rules or metric-based zones.

Methodology
The authors employ a Multi-Agent Reinforcement Learning (MARL) framework to derive flocking dynamics from first principles.

Agent Dynamics: $N$ agents move in a 2D unbounded space with constant speed $v_0$ . Agents perceive only their topological (Voronoi) neighbors. Their motion is updated by choosing a new heading that is a weighted combination of the mean neighbor orientation ( $V_i$ ) and the direction toward the local center of mass ( $R_i$ ).
Control Parameter: The relative weight $\beta \in [0, 1]$ between alignment and attraction is the learnable action. $\beta=0$ corresponds to pure alignment (Vicsek-like), while $\beta=1$ corresponds to pure attraction.
Learning Framework: The study utilizes two training paradigms:
1. Centralized Training (CT): Agents share a single Q-matrix and learn from a shared experience pool.
2. Decentralized Training (DT): Each agent maintains its own Q-matrix and learns independently based on its specific trajectory.
Cost Function: Agents are trained to minimize a cost function $L(d)$ based on the mean distance $d$ to their neighbors. The function is a non-convex, Lennard-Jones-like potential: $L(d) = a/d - b/\sqrt{d}$ . This function penalizes both excessive separation ( $d \to \infty$ ) and close-range crowding ( $d \to 0$ ), with a minimum at an optimal distance $\bar{d}=1$ .
Optimization: The agents use a Q-learning algorithm with $\epsilon$ -greedy exploration to find the optimal policy $\beta^*(d)$ that minimizes the cumulative cost over time.

Key Contributions and Results

Emergence of Cohesive Flocking: The study demonstrates that agents trained with a cost function penalizing both large separations and small distances spontaneously learn a policy that yields cohesive flocking with high polar order ( $\Phi \approx 0.95$ ). This behavior is robust across both CT and DT paradigms for moderate group sizes ( $N < 800$ ).
Role of Collision Avoidance: A critical finding is that the penalty for small distances is essential for flocking. When the cost function is modified to remove the divergence at small $d$ (effectively removing the collision penalty), the agents learn a strategy dominated by attraction ( $\beta \approx 1$ ), resulting in a disordered, cohesive swarming state rather than flocking. This suggests that the need to avoid collisions is a primary driver for the emergence of alignment-based flocking.
Distance-Dependent Policy: Analysis of the learned Q-matrices reveals a distinct regime split:
- Short Distances ( $d < d^*$ ): The optimal policy is dominated by strong alignment ( $\beta \approx 0$ ).
- Large Distances ( $d > d^*$ ): The policy involves a flexible combination of alignment and attraction. The authors show via "policy distillation" that the exact balance in this regime is not critical; as long as some attraction exists, the system maintains cohesion.
Structural Characteristics: The resulting flocks exhibit structural features consistent with biological observations (e.g., starling murmurations). They behave as a "structureless nonequilibrium liquid" with a finite but low neighbor exchange rate (mixing rate) and a radial pair distribution function $g(r)$ lacking the oscillatory structure of typical liquids, matching experimental data.
Scalability and Robustness: The algorithm scales effectively up to $N=1600$ agents under centralized training. Decentralized training shows reduced performance for very large groups ( $N > 800$ ) due to the difficulty of individual agents exploring the full state space within finite training time, though this can be mitigated by adjusting exploration exponents.
Boundary Conditions: The authors tested an alternative cost function based on Voronoi cell area (to distinguish interior vs. boundary agents) and found no qualitative difference in the resulting flocking behavior compared to distance-based costs.

Significance and Claims
The paper claims to resolve the question of why certain gregarious species exhibit flocking rather than swarming. The authors argue that flocking emerges not merely as a mechanism for group cohesion, but as an optimal strategy that simultaneously satisfies the imperative to stay together and the imperative to avoid dangerous collisions. The learned dynamics are robust to noise and algorithmic details, suggesting that the interplay between alignment and attraction, modulated by distance-dependent collision avoidance, is a sufficient condition for the spontaneous emergence of complex collective motion. The results are presented as biologically plausible for real birds (due to the quasi-2D nature of flight) and potentially applicable to swarm robotics, offering a learning-based alternative to pre-programmed interaction rules.