🤖 machine learning

Dimensionality Controls When Modularity Helps in Continual Learning

This paper demonstrates that the effectiveness of modular architectures in continual learning depends on representational dimensionality, as modularity only provides a decisive advantage in low-dimensional "rich" regimes by enabling adaptive, task-dependent subspace organization, whereas in high-dimensional "lazy" regimes, explicit modularity offers little benefit over single-network baselines.

Original authors: Kathrin Korte, Christian Medeiros Adriano, Joachim Winther Pedersen, Eleni Nisioti, Sebastian Risi

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kathrin Korte, Christian Medeiros Adriano, Joachim Winther Pedersen, Eleni Nisioti, Sebastian Risi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning to play two different instruments: the piano and the violin.

The Big Problem: The "Sticky" Brain
When you learn a new skill, your brain (or a computer program) faces a tricky dilemma. It needs to be plastic (flexible enough to learn the new violin) but also stable (holding onto the piano skills so you don't forget them). If you try to learn the violin using the exact same mental "muscle" as the piano, you might accidentally overwrite your piano knowledge. This is called the "stability-plasticity dilemma."

The Proposed Solution: Building Separate Rooms
One way to solve this is to build a modular system. Instead of one big brain, you give the learner two separate "rooms" (modules). One room is for piano, the other for violin. The idea is that by keeping them physically separate, the piano skills can't get messed up by the violin lessons.

The Paper's Discovery: It Depends on the Size of the Room
The researchers in this paper asked: Does building separate rooms always help?

They found the answer is no. It depends entirely on how much "space" the learner has to work with. They tested this by changing the "size" of the mental space (which they call dimensionality).

Here is the breakdown using a simple analogy:

1. The "Huge Warehouse" Scenario (High Dimensionality)

Imagine the learner has a massive, empty warehouse with infinite shelves.

What happens: Whether you build two separate rooms or just use one giant open floor plan, it doesn't matter much. There is so much space that the piano and violin skills can just sit on different shelves without bumping into each other.
The Result: The "separate rooms" (modular) design performs just as well as the "one big room" (single network) design. The extra structure of having separate rooms is wasted because there's plenty of room to spread out anyway.

2. The "Tiny Studio Apartment" Scenario (Low Dimensionality)

Now, imagine the learner is forced into a tiny, cramped studio apartment. There is very little space.

What happens: If you try to fit piano and violin skills into one tiny room, they will clash. You might put your violin sheet music on the piano keys, and suddenly you can't play the piano anymore. This is interference.
The Result: This is where the separate rooms (modular) design shines.
- Because the space is so tight, the modular design forces the system to organize itself smartly.
- If the tasks are very similar (like piano and organ), the system puts them in overlapping corners of the room because they share tools.
- If the tasks are somewhat different, they get their own distinct zones.
- If the tasks are totally different (piano and juggling), they get pushed into completely opposite corners to avoid crashing into each other.

The Key Takeaway

The paper claims that modularity (separate rooms) is only helpful when the "space" is tight.

In a big space: You don't need separate rooms; a single big room works fine.
In a small space: Separate rooms are essential to keep things organized and prevent chaos.

The "Graded" Organization

The most interesting finding is that in the "small space" scenario, the modular system didn't just separate everything blindly. It created a smart, graded map:

Similar tasks overlapped (sharing space).
Different tasks stayed apart (separating space).
This created a very organized, efficient layout that the single-network system couldn't achieve on its own.

Summary

Think of it like packing for a trip.

If you have a giant moving truck, you can throw everything in one big pile, and it doesn't matter much how you organize it.
If you only have a small backpack, you must organize carefully. You need to know that socks go with shoes, and fragile items go in their own pockets.

This paper shows that modular architecture is like the careful packing strategy. It's not always necessary (when you have a giant truck), but when you are constrained (in a small backpack), it is the only way to keep your skills from getting mixed up and destroyed.

Technical Summary: Dimensionality Controls When Modularity Helps in Continual Learning

Problem Statement

Continual learning faces the fundamental "stability–plasticity dilemma": a system must acquire new knowledge (plasticity) while preserving previously learned representations (stability). This tension is exacerbated when tasks share structure, creating a risk of interference where new learning disrupts prior knowledge. While modular architectures are often proposed to mitigate interference by isolating task-specific computations, it remains unclear when such structural biases are functionally beneficial versus when they are redundant. Specifically, the paper investigates whether the utility of modularity depends on the representational regime—the geometric properties of the learned representations, particularly their effective dimensionality.

Methodology

Experimental Paradigm

The study utilizes a sequential A–B–A transfer–interference paradigm (inspired by Holton et al., 2026).

Phase A1: The network learns Task A (mapping plant cues to dial positions based on a seasonal rule).
Phase B: The network learns Task B on new stimuli. Task B shares the same formal structure as A but varies in rule similarity: Same (identical rule), Near (small angular shift), or Far (large angular shift).
Phase A2: The network is retested on Task A to measure retention and interference.

Architectures

Two recurrent architectures were compared:

Single-Network Baseline: A single recurrent population processes all inputs with a shared readout.
Task-Partitioned Modular Network: Inputs are routed to distinct recurrent modules ( $M_A$ for Task A, $M_B$ for Task B) based on task identity. Module outputs are concatenated and processed by a shared readout. This enforces structural separation in recurrent processing but maintains coupling at the output level.

Controlling Representational Regimes

To manipulate the effective dimensionality of the learned representations without changing network capacity, the authors varied the initialization weight scale ( $\gamma$ ).

High-Dimensional ("Lazy") Regime: Large $\gamma$ values keep representations close to initialization, resulting in high effective dimensionality and less specialized codes.
Low-Dimensional ("Rich") Regime: Small $\gamma$ values force representations into lower-dimensional manifolds, inducing structured, specialized codes.
Metric: Effective dimensionality was quantified as the number of principal components required to explain 99% of the variance in hidden states.

Analysis

Performance was measured via angular accuracy, transfer (improvement on Task B based on Task A), and interference (shift in Task A behavior toward Task B rules). Representational geometry was analyzed using Principal Component Analysis (PCA) to compute effective dimensionality, principal angles between task-specific subspaces, and 3D visualizations of task trajectories.

Key Results

1. Modularity is Conditional on Dimensionality

The benefit of modular architecture is not universal but gated by the representational regime:

High-Dimensional (Lazy) Regime: Both single and modular networks achieved comparable accuracy, transfer, and interference levels. The explicit structural separation of the modular network provided no significant advantage, as the high-dimensional space allowed the single network to encode tasks with limited geometric pressure.
Low-Dimensional (Rich) Regime: Modularity became decisive. The modular network maintained high Task A accuracy and low interference across all similarity conditions. In contrast, the single network suffered significant interference, particularly in the "Far" condition, as it struggled to reorganize its constrained representational space without overwriting prior knowledge.

2. Emergence of Similarity-Dependent Geometry

In the low-dimensional regime, the modular network developed a graded, compositional organization of representational subspaces that the single network failed to achieve:

Similar Tasks (Same): Representations remained closely aligned (overlapping subspaces).
Moderately Dissimilar Tasks (Near): Representations occupied intermediate angles (partial separation).
Dissimilar Tasks (Far): Representations became strongly orthogonal (separated subspaces).
The single network exhibited a weaker, less structured dependence on task similarity, with subspaces remaining more entangled regardless of task distance.

3. Robustness of Findings

Ablation studies varying module size and input routing confirmed that these effects are robust to moderate architectural variations. The core finding—that dimensionality gates the functional impact of modularity—held across configurations.

Key Contributions

Identification of a Gating Variable: The paper identifies the initialization-induced representational regime (co-varying with dimensionality) as the key factor determining when structural modularity becomes functionally beneficial in continual learning.
Characterization of Compositional Geometry: It demonstrates that in constrained (low-dimensional) regimes, modular networks naturally develop a similarity-dependent geometric organization (overlapping $\to$ partial $\to$ orthogonal) that supports compositional reuse while minimizing interference.
Reframing Continual Learning Goals: The findings support viewing continual learning not as a binary choice between "sharing" and "separation," but as an adaptive allocation of representational subspaces where the optimal geometry depends on task similarity.

Significance and Claims

The paper argues that representational dimensionality controls when modularity helps. Structural biases (like modularity) only reshape learning behavior and internal geometry when the representational space is sufficiently constrained to make geometry a binding factor. In high-dimensional "lazy" regimes, the network has enough degrees of freedom to solve tasks without needing explicit structural separation. However, in low-dimensional "rich" regimes, the network is forced to organize computations into distinct subspaces, at which point architectural modularity provides a crucial advantage in managing the stability-plasticity trade-off.

The authors conclude that robust compositional continual learning requires mechanisms that dynamically regulate both representational dimensionality and subspace organization in response to changing task structures, rather than relying on fixed architectural separation alone.

Note: The paper explicitly states limitations, including the use of PCA as a proxy for intrinsic dimensionality and the restriction to short, controlled task sequences, acknowledging that further work is needed to determine if these effects extend to longer, more heterogeneous continual learning benchmarks.