🤖 machine learning

Fluid-Agent Reinforcement Learning

This paper introduces a "fluid-agent" framework for multi-agent reinforcement learning that enables agents to dynamically spawn new agents, proposing game-theoretic solutions and demonstrating through experiments that this approach allows teams to adapt their size to environmental demands and discover novel strategies beyond fixed-population settings.

Original authors: Shishir Sharma, Doina Precup, Theodore J. Perkins

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shishir Sharma, Doina Precup, Theodore J. Perkins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a video game. In most standard games, the rules are simple: you start with a fixed number of characters (like a team of four soldiers), and you have to win the level with exactly those four. If you run out of ammo or get hurt, you can't just call for backup; you have to make do with what you have.

This paper introduces a new way of thinking about Artificial Intelligence (AI) called Fluid-Agent Reinforcement Learning.

Here is the simple breakdown of what the authors are doing, using some everyday analogies.

1. The Core Idea: The "Self-Replicating" Team

In the real world, teams aren't static.

In a business: A successful company might decide to spin off a new division or hire more staff to handle a sudden rush of orders.
In nature: A single cell divides into two, or a colony of ants grows as food becomes available.

The authors ask: What if our AI agents could do the same thing? Instead of being stuck with a fixed team size, what if an AI agent could say, "This job is too hard for me alone; I'll create a helper," or "We have too many people here; I'll stop making new ones"?

They call this a Fluid-Agent Environment. The "fluid" part means the number of players on the team can change dynamically, just like water flowing into a cup or draining out.

2. The Rules of the Game (The Framework)

The authors created a mathematical framework (called POFSG) to study this. Think of it as the rulebook for a new type of game.

The Spawn Button: Every agent has a special button. If they press it, a new agent appears on the map.
The Cost: Creating a new agent isn't free. It costs "energy" (or points). If you spawn too many, you might run out of resources.
The Goal: The team wants to win the game (get the highest score), but they have to balance the cost of making new friends against the benefit of having more hands to do the work.

3. The Experiments: Three New Playgrounds

To test this idea, the researchers built three different "sandboxes" (environments) where agents had to learn how to manage their team size.

A. Predator-Prey (The Wolf Pack)

The Setup: Blue wolves (agents) try to catch red rabbits. To catch a rabbit, two wolves must surround it.
The Twist: If there are too few wolves, they can't catch anything. If there are too many, they waste energy running around.
The Lesson: The AI learned to act like a smart pack leader. When there were lots of rabbits, the wolves spawned more helpers. When rabbits were scarce, they stopped spawning to save energy. They learned to size their team to match the food supply.

B. Level-Based Foraging (The Construction Crew)

The Setup: Agents are construction workers with different "levels" (strength). They need to collect apples. A level-5 apple requires a team with a total strength of 5.
The Twist: When a worker spawns a new one, the new worker inherits the parent's strength.
The Lesson: The AI learned not just how many to spawn, but what kind to spawn. If they needed to lift a heavy box, they didn't just spawn a weak helper; they spawned a strong one. They learned to build the perfect team composition.

C. PuddleBridge (The Human Tower)

The Setup: Agents need to cross a wall. There is a "gate" that sometimes opens (easy path) and sometimes closes (hard path). If the gate is closed, they must cross a swamp.
The Mechanic: To cross the swamp, one agent must stand in the water, and another must stand on top of them (like a human tower) to jump to the other side.
The Lesson: This was the most complex test. The AI had to learn a "switch."
- Gate Open? "I'll go alone. No need to spawn."
- Gate Closed? "I need a partner! I'll spawn one, we'll stack up, and cross together."
- They learned to switch between being a solo act and a team player depending on the situation.

4. Why This Matters (The "So What?")

Most current AI research assumes the number of players is fixed. This paper argues that's too limiting.

Realism: In the real world, populations change. Companies grow and shrink; cells divide; traffic jams form and dissolve.
Efficiency: By letting AI decide its own team size, it becomes much more efficient. It doesn't waste resources on unnecessary workers, and it doesn't get overwhelmed by having too few.
New Strategies: It unlocks strategies that are impossible in fixed teams. For example, in the "PuddleBridge" game, the AI discovered a strategy of "stacking" that only works because it could dynamically create a second agent on the fly.

The Big Takeaway

Imagine a video game where you don't just control one character. You control a leader who can decide to clone themselves, hire a friend, or fire a teammate, all while trying to win the level.

This paper proves that AI can learn to make those decisions. It shows that when you give AI the freedom to change its own team size, it becomes smarter, more adaptable, and better at solving complex, real-world problems where the number of people involved is never the same twice.

1. Problem Statement

Traditional Multi-Agent Reinforcement Learning (MARL) assumes a fixed number of agents interacting within an environment. While mathematically convenient, this assumption fails to capture real-world dynamics where the number of agents is variable and often determined by the agents themselves (e.g., cell division, corporate spin-offs, or unit creation in strategy games).

The paper addresses the gap in formalizing environments where:

The population size is not fixed and can change over time.
Agents possess the agency to create (spawn) new agents, thereby introducing new decision-making entities into the system.
Existing solution concepts (like Nash Equilibrium) and algorithms need to be re-evaluated in the context of dynamic population sizes and the associated credit assignment and exploration challenges.

2. Methodology

2.1 Theoretical Framework: POFSG

The authors formalize this setting using Partially Observable Fluid Stochastic Games (POFSG). This extends the standard Partially Observable Stochastic Game (POSG) tuple $\mathcal{G} = \langle I, S, L, A, O, T, Z, R, \gamma \rangle$ with a key addition:

$L: S \to 2^I$ (Alive Function): A function that maps a state to the set of currently alive agents.
Dynamic Action Space: The joint action space $A$ varies based on $L(s)$ . Crucially, the action space includes a "spawn" action.
Population Ceiling: A maximum population size $N$ is enforced. If the population is at capacity, spawning has no effect.

2.2 Game-Theoretic Analysis

The authors prove the existence of standard solution concepts within this fluid framework:

Nash Equilibrium (NE): They prove that every POFSG possesses a stationary mixed-strategy Nash equilibrium. The proof leverages Fink's proof for stochastic games by treating "unborn" agents as having a "Do Nothing" action, effectively transforming the fluid game into a fixed-player game with state-dependent action sets.
Subgame-Perfect Nash Equilibrium (SPNE): They prove the existence of SPNE for finite-horizon POFSGs with public joint actions and perfect recall. This is achieved by sequentializing the simultaneous-move game into an extensive-form game and applying backward induction.

2.3 Experimental Environments

To evaluate the framework, the authors modified established benchmarks and introduced a new environment:

Fluid Predator-Prey: Predators must cooperate to capture prey. Agents can spawn new predators.
- Reward Variants: Size-Constant Payoff (SCP) (total reward scales with population) vs. Size-Inverse Payoff (SIP) (per-agent reward decreases as population grows).
Fluid Level-Based Foraging (LBF): Agents collect food requiring specific sum of agent levels. Spawned agents inherit the parent's level. This tests adaptive team composition (optimizing what to spawn, not just how many).
PuddleBridge (New): A grid world with a wall barrier.
- Mechanic: Agents can stack in "puddles" to form a bridge. The bottom agent is locked; the top agent moves.
- Goal: Reach a goal on the other side. If a gate is open, one agent suffices; if closed, two agents (stacked) are required. This tests the ability to switch between fluid (spawn) and non-fluid (fixed) strategies based on context.

2.4 Exploration Strategy

To address the difficulty of exploring variable population sizes, the authors employ Curriculum Learning:

Randomly sampling initial population sizes and population ceilings during training.
Introducing a specific exploration parameter $\epsilon_{spawn}$ that increases linearly, encouraging agents to try spawning early in training.

3. Key Contributions

Formal Definition of Fluid-Agent RL: The first rigorous mathematical framework (POFSG) for MARL where agents can dynamically alter the population size.
Theoretical Guarantees: Proof of the existence of NE and SPNE in fluid-agent games, validating the use of standard game-theoretic concepts.
Algorithmic Insights:
- Demonstrated that Joint Return Optimizers (e.g., VDN, MAPPO) outperform Per-Agent Optimizers (e.g., IQL) in fluid settings, particularly when reward structures incentivize population growth.
- Showed that agents can learn to optimize team composition (e.g., spawning specific agent levels in LBF) rather than just population size.
- Revealed that fluidity allows agents to discover heterogeneous policies (switching between single-agent and multi-agent strategies) that are inaccessible in fixed-population settings.
New Benchmark: Introduction of PuddleBridge, a novel environment specifically designed to test dynamic strategy switching and spatial coordination via agent spawning.

4. Results

Algorithm Performance:
- In Predator-Prey with Size-Inverse Payoff (where spawning reduces per-agent reward), per-agent optimizers (IQL, PPO) failed to spawn, while joint optimizers (VDN) successfully balanced costs and benefits, converging on an optimal population size (~6 agents).
- In Size-Constant Payoff, joint optimizers successfully maximized the joint return by spawning the maximum number of agents, whereas per-agent optimizers struggled to coordinate this global optimum.
Adaptability to Task Variability:
- In the Ablation Study, fluid groups dynamically adjusted their population size based on prey availability. They performed competitively across all resource levels, whereas fixed groups either under-performed in high-resource settings (too few agents) or high-cost settings (too many agents).
Role Optimization:
- In LBF, algorithms like VDN and MAPPO learned to spawn exactly one additional agent of a specific level (Level 2) to minimize costs while maximizing food collection capabilities, proving agents can optimize characteristics of spawned units.
Strategy Switching:
- In PuddleBridge, agents learned to spawn only when the "gate" was closed (requiring a bridge) and refrain from spawning when the gate was open (where a single agent is optimal). This demonstrated the ability to fluidly switch between fixed and fluid strategies.

5. Significance and Future Work

Significance:
This work bridges the gap between theoretical MARL and complex real-world systems where population dynamics are endogenous. It demonstrates that allowing agents to control their own numbers leads to more robust, adaptable, and efficient solutions. It challenges the standard assumption of fixed team sizes and provides a new lens for studying coordination, credit assignment, and emergent behavior.

Future Directions:

Extending the framework to include agent death (not just creation).
Investigating theoretical properties such as sample complexity and equilibrium convergence rates in fluid systems.
Applying these concepts to more complex real-world domains like biological modeling, autonomous logistics, and economic markets.