Insect-inspired modular architectures as inductive biases for reinforcement learning

This paper demonstrates that an insect-inspired modular RL architecture, which decomposes control into specialized interacting circuits rather than a single centralized controller, outperforms monolithic MLP and GRU models in complex navigation tasks involving competing behavioral objectives.

The Gist

The "Swiss Army Knife" vs. The "Specialized Crew"

Imagine you are trying to teach a robot how to navigate a busy park. The robot needs to find a snack (food), avoid bumping into benches (obstacles), and run away if a big, scary dog (a predator) starts chasing it.

Currently, most AI researchers use a "Single Brain" approach. They give the robot one big, massive brain (a neural network) and tell it: "Here is everything you see; now, decide exactly how much to move your left leg and right leg." This brain tries to do everything at once—it’s trying to remember where the food was, keep track of its direction, and watch out for the dog—all inside one giant, messy soup of information.

This paper suggests a different way: The "Specialized Crew" approach, inspired by how insects (like ants or bees) actually work.

---

The Insect Strategy: A Team of Experts

Instead of one giant brain, the researcher (A.E. Staples) built a robot with a "distributed" system. Think of it like a small, highly efficient crew working in a command center:

1. The Scouts (Sensory Encoding): Instead of one eye, you have different specialists. One scout only watches for obstacles, another only tracks the food, and another only feels how the robot is moving. They don't get confused by each other's jobs.
2. The Compass (Heading State): This is a tiny, dedicated module that does nothing but keep track of which way the robot is facing. It’s like having a built-in GPS that never forgets North.
3. The Librarian (Associative Memory): This module acts like a quick-access filing cabinet. It stores "if-then" memories (e.g., "Last time I saw something moving fast, it was a predator") without cluttering up the rest of the brain.
4. The Captain (Command Center): This is the boss. The Captain listens to the Scouts and the Librarian and decides the vibe of the mission. Is it "Search Mode"? "Panic Mode"? "Steady Walking Mode"?
5. The Specialists (Local Controllers): This is the coolest part. Instead of one motor controller, there is a team of experts: an "Avoidance Expert," a "Food-Seeker Expert," and a "Stabilizer Expert."
6. The Referee (The Arbiter): The Referee listens to the Captain and decides which Expert gets to drive the robot right now. If a dog is chasing you, the Referee gives 99% of the control to the "Avoidance Expert" and tells the "Food-Seeker" to sit down and be quiet.

---

Why does this work better?

In the experiments, the "Single Brain" robots struggled. The "All-in-One" brain often got overwhelmed. It would try to look for food and run from the dog at the same time, resulting in a confused, wobbly robot that often crashed.

The Insect-Inspired Crew, however, was much more decisive. Because the "Avoidance Expert" was a specialist, it could take over instantly when the predator appeared. The paper found that this modular approach:

• Got higher scores: It found food and survived longer.
• Was more stable: It didn't "glitch out" or have mental breakdowns during training like the single-brain models did.
• Was more decisive: The "Referee" learned to pick one expert at a time very clearly, rather than trying to listen to everyone at once.

The Big Picture

The takeaway isn't just that "insects are smart." The takeaway is that structure matters.

When a problem is complicated and requires switching between different goals (like "relax" vs. "run for your life"), it is much more efficient to break the brain into specialized departments rather than trying to cram every single thought into one giant, crowded room. By mimicking the "modular" design of nature, we can build AI that is more organized, more decisive, and better at handling the chaos of the real world.

Technical Summary

Technical Summary: Insect-Inspired Modular Architectures as Inductive Biases for Reinforcement Learning

Problem Statement

Modern Reinforcement Learning (RL) for continuous control typically relies on centralized architectures (e.g., MLPs or GRUs). These models compress all sensory observations, temporal memory, and action selection into a single, monolithic latent state. While flexible, this approach lacks the structural specialization found in biological nervous systems. The author investigates whether architectural decomposition—specifically inspired by the distributed neural circuits of insects—can serve as a superior inductive bias for RL agents facing tasks with dynamically competing objectives (e.g., seeking food while simultaneously avoiding obstacles and predators).

Methodology

The study compares an insect-inspired modular controller against two centralized baselines: a Multilayer Perceptron (MLP) and a Gated Recurrent Unit (GRU).

1. The Task:
A 2D navigation environment requiring the agent to balance three competing goals: food seeking, obstacle avoidance, and predator escape. The environment is partially observed and continuous.

2. The Modular Architecture:
Instead of a single bottleneck, the proposed model decomposes control into five functional modules:

• Sensory Encoding: Separate MLPs for vision-like, proprioceptive, and task-specific features, fused into a shared representation.
• Heading State: A recurrent state using a ring-structured recurrent kernel, mimicking the population codes of the insect central complex to maintain directional persistence.
• Associative Memory: A sparse, high-dimensional projection inspired by the mushroom body (Kenyon-cell coding) to support sparse memory retrieval.
• Command Center: A GRU-based module that integrates sensory input, heading, and memory to produce behavioral mode probabilities ($\pi_t$) and low-dimensional command signals ($c_t$).
• Local Controllers & Arbiter: A bank of four specialized controllers (stabilize, avoid, approach, explore) that propose candidate actions. A learned arbiter uses a softmax mechanism to assign motor authority (weights) to these modules.

3. Training:
The models were trained using Proximal Policy Optimization (PPO). The modular model included specific auxiliary loss terms to encourage economical command signals, diffuse behavioral mode probabilities, and sparse module selection (via module-assignment entropy).

Key Contributions

• Bio-inspired Inductive Bias: Demonstrates that mimicking the functional separation of insect neural structures (central complex and mushroom body) provides a computational advantage in RL.
• Structural Factorization: Proposes a method to factorize a policy into interacting state variables with distinct roles (directional state, sparse memory, command manifold, and arbitration weights) rather than a single latent vector.
• Decoupled Entropy Management: Shows that an architecture can maintain high-level behavioral diversity (high mode entropy) while achieving highly decisive, specialized motor control (low module entropy).

Results

In a six-seed experiment over 75 PPO updates, the modular controller significantly outperformed the baselines:

• Performance: The modular policy achieved a final mean episodic return of $-2798.8 \pm 964.4$, outperforming the GRU ($-3778.0 \pm 628.1$) and the MLP ($-4727.5 \pm 772.5$).
• Stability: The modular model and GRU showed stable optimization (low KL divergence and clip fractions). In contrast, the MLP was highly unstable, with a clip fraction of $\approx 0.97$, suggesting it was ill-suited for the task's complexity.
• Control Selectivity: The modular controller achieved a very low module-assignment entropy ($0.0457 \pm 0.0244$), proving it learned to selectively delegate authority to specific modules based on the environmental context.
• Value Accuracy: The modular model achieved the lowest final value loss, indicating more accurate critic predictions.

Significance

The paper concludes that distributed modular control is systematically advantageous for tasks involving dynamically competing behavioral objectives. The advantage stems from the architecture's ability to reduce "interference" between incompatible drives by factorizing the policy. This suggests that biological structures are not just "realistic" but are mathematically meaningful decompositions that can guide the design of more efficient and robust machine learning controllers.