🤖 machine learning

Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning

This paper demonstrates that deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations that scale with task complexity and effectively bound achievable performance, revealing the intrinsic dimensionality of the underlying problem without explicit sparsity-inducing objectives.

Original authors: Scott M. Norton

Published 2026-07-30

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Scott M. Norton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers learn to play games not by reading a manual, but by staring at a screen and guessing what to do next. This is the realm of Deep Reinforcement Learning (DRL). Think of it like training a dog: you don't tell the dog how to catch a frisbee; you just give it a treat when it does, and a "nope" when it doesn't. Over time, the dog figures out the trick. In the computer world, the "dog" is a neural network—a giant web of mathematical connections that mimics a brain. Usually, to make these computers smart, we let them tweak every single connection in the web, hoping they figure out the best way to see the world. But here's the catch: these computer brains are often massively overgrown. They have millions of connections, like a forest with too many trees, making it hard to see which ones are actually doing the work and which are just getting in the way. Scientists have long wondered: do these giant brains need to be so huge, or is there a simpler, more efficient way to solve these problems?

This is where a curious researcher named Scott M. Norton stepped in with a very strange experiment. Instead of letting the computer's "eyes" (the part that looks at the screen) learn and change, he froze them. He took a random, untrained camera lens and locked it in place, refusing to let it learn anything new. He then asked the computer to learn how to play Atari games using only this frozen, random lens. The expectation was that the computer would fail miserably, or at least struggle to make sense of the blurry, random images. But something magical and unexpected happened. The computer didn't just learn; it learned to be incredibly efficient. Without anyone telling it to be simple, it spontaneously decided to ignore 95% of its own brainpower. It turned off almost all its neurons and solved the game using only a tiny handful of them. It was as if a student, handed a textbook with random pages glued together, managed to ace the exam by memorizing just three sentences.

The Frozen Lens Experiment

In this study, the researcher set up a classic video game scenario using the game Pong (the digital tennis game where you hit a ball back and forth). He built a standard AI agent, but with a twist: the part of the AI that processes the video images (the Convolutional Neural Network, or CNN) was initialized with random numbers and then frozen. It was never allowed to change. The only part of the AI that could learn was the final decision-making layer, the "fully connected" part that takes the image data and decides whether to move the paddle up, down, or stay still.

Usually, when we train these AI agents, we let the whole system learn together. The "eyes" learn to spot the ball, and the "brain" learns to move the paddle. But in this experiment, the "eyes" were stuck with a random, untrained view of the world. The researchers expected the AI to be clumsy. Instead, they found that the AI's brain spontaneously developed a sparse representation.

"Sparsity" is a fancy word for "using very little." Imagine you have a room with 64 light switches. In a normal, fully trained AI, almost all 64 switches would be flicking on and off as it plays the game, creating a complex, noisy pattern. But in this frozen experiment, the AI turned on only 1 to 3 switches out of the 64 available. It ignored the other 61 switches completely. It didn't need them. The AI figured out that the random, frozen lens happened to have just the right few "channels" of information to solve the game, so it turned off everything else to save energy and focus.

The "Just Enough" Rule

The most fascinating part of the discovery is that the number of switches the AI used wasn't random; it matched the difficulty of the game.

Pong: This is a simple game with just a ball and two paddles. The AI needed only 1 to 3 neurons (the tiny processing units in the brain) to master it.
Breakout: This game involves a ball, a paddle, and a wall of bricks. It's more complex. The AI needed 19 to 26 neurons.
Space Invaders: This is a chaotic game with aliens, bullets, and shields. The AI needed about 42 neurons.

The researchers tested this by making the AI's brain wider (giving it more switches to choose from). Even when they gave the AI 128 switches instead of 64, it still only used about the same small number (3 to 14) to solve the game. It didn't use more just because it had them available. It seems the AI was sensing the "true complexity" of the game and only using as much brainpower as was strictly necessary.

Why This Matters (And What It Isn't)

You might wonder, "Did the AI just get lucky?" The researchers checked this thoroughly. They found that if they took the specific few neurons the AI was using and turned them off (a process called ablation), the AI instantly forgot how to play and started acting completely randomly. This proved that those few neurons were doing all the heavy lifting. The rest of the brain was truly useless for that specific task.

However, the paper is careful not to claim this is a magic bullet for every situation. The researchers found that this "frozen lens" trick works best when the game has a clear, reactive structure (like Pong or Breakout). In some cases, like the game Freeway, the AI could find a "cheat" by just moving up every time, which didn't require any real learning. In those cases, the "sparsity" wasn't a sign of intelligence; it was a sign of a shortcut. But for the games where real skill was needed, the frozen random lens forced the AI to find the most efficient path possible.

The Big Picture: Finding the Signal in the Noise

The story here is about efficiency. We often think that to solve a hard problem, you need a massive, complex machine. But this paper suggests that if you give a machine a fixed, random set of tools, it will naturally figure out the simplest way to use them. It's like giving someone a giant toolbox with a million random tools. Instead of trying to use them all, a smart person will quickly realize, "Oh, I only need this one screwdriver and this one hammer to fix this chair."

The researchers also noticed something about when this happens. The AI decided which neurons to use very early in its training—sometimes within the first 15 million steps. It locked in those choices long before it actually started getting good at the game. It was as if the AI picked its team members on day one, and then spent the next 100 million steps just practicing with that specific team.

A Playful Analogy: The Radio Tuner

Imagine the frozen CNN is like a radio that is stuck on a specific, random frequency. The signal coming in is a mix of static and music. Most people would think, "This radio is broken; I need to fix the antenna." But this AI is different. It doesn't try to fix the antenna. Instead, it listens to the static and realizes, "Hey, if I just tune into this one tiny slice of the static, I can hear the music perfectly." It ignores the rest of the noise.

In the world of AI, we usually try to build better antennas (train the whole network). This paper shows that sometimes, if you just let the AI listen to a random antenna, it will naturally find the one tiny slice of the signal that matters and ignore the rest. It's a reminder that sometimes, less is more, and that nature (or in this case, a frozen random network) has a way of finding the simplest solution if we just stop getting in its way.

The researchers suggest that this phenomenon might help us understand how to build smarter, smaller AI systems in the future. Instead of building giant, wasteful networks, maybe we can use frozen, random features to help AI find the "intrinsic dimension" of a problem—the true, small number of variables that actually matter. It's a step toward making AI that is not just powerful, but also elegant and efficient.

Technical Summary: Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning

Problem Statement

Deep Reinforcement Learning (DRL) agents typically employ overparameterized neural networks to solve environments governed by a small number of intrinsic state variables (e.g., Pong is defined by roughly five continuous variables). This overparameterization creates a tension between capacity and structure, often leading to pathologies such as representation rank collapse, dormant neurons, and plasticity loss. While sparsity is often viewed as a desirable property to be engineered via pruning or regularization, existing literature suggests that sparsity is not naturally emergent in standard end-to-end training; rather, dense networks distribute information uniformly across available capacity, obscuring the underlying simplicity of the task.

This work investigates a complementary question: How little does gradient descent choose to use when the feature extractor is fixed? Specifically, the authors examine whether sparsity emerges spontaneously when an agent must learn a policy using a frozen, randomly initialized Convolutional Neural Network (CNN) as a feature extractor, without any explicit sparsity-inducing objectives.

Methodology

Architecture and Training Protocol

The study utilizes the standard Nature-DQN CNN architecture (three convolutional layers producing 3,136-dimensional features) followed by two fully connected (FC) layers (FC1: 3,136 $\to$ 64; FC2: 64 $\to$ 64) and policy/value heads.

Frozen-CNN Condition: The CNN weights are initialized randomly (Kaiming normal) and immediately frozen. Only the FC layers and policy/value heads are trainable. The CNN acts as a fixed random projection from pixel space to a 3,136-dimensional feature space.
Trainable-CNN Condition (Control): All layers are trained end-to-end with identical hyperparameters.
Algorithms: Experiments use Proximal Policy Optimization (PPO) across two independent implementations: Stable-Baselines3 (SB3, synchronous) and Sample Factory (SF, asynchronous).
Environments: Four Atari 2600 games spanning a range of intrinsic complexity: Pong ( $\sim$ 5 state variables), Freeway ( $\sim$ 10–15), Breakout ( $\sim$ 15–25), and Space Invaders ( $\sim$ 40+).
Stochasticity: "Sticky actions" (repeating the previous action with probability $p_{sticky}=0.25$ ) are employed where appropriate to prevent memorization of fixed action sequences and ensure reactive control.

Analysis Metrics

Active Neuron Count: A neuron is classified as "active" if its mean post-ReLU activation exceeds 0.01.
Participation Ratio (PR): A measure of effective dimensionality defined as $PR = (\sum \lambda_i)^2 / \sum \lambda_i^2$ , where $\lambda_i$ are activation-covariance eigenvalues. This distinguishes between a set of active neurons that are independent versus highly correlated.
Ablation Verification: The critical empirical test involves masking active neurons (FC1-Remove) or inactive neurons (FC1-Keep). A crash to random play upon removing active neurons confirms a genuine information bottleneck.

Key Contributions and Results

1. Emergent Sparsity Scales with Task Complexity

The primary finding is that frozen random CNNs spontaneously develop extremely sparse representations in the first fully connected layer (FC1), and the number of active neurons correlates with task complexity:

Pong (Deterministic): 1–3 active neurons out of 64 (1.6–4.7% utilization).
Breakout: 19–26 active neurons.
Space Invaders: $\sim$ 42 active neurons.
Trainable Controls: In contrast, trainable CNNs utilize 55–64 neurons (91–100% utilization) regardless of task complexity.
Width Scaling: Experiments varying FC1 width (32, 64, 128) confirm that the absolute active count is stable and reflects task structure, not a fixed fraction of network capacity.

2. Capacity-Limited Scaling Within Games

Within the same game (Pong) and matched hyperparameters, three independent random projections (seeds) produced different active neuron counts (5, 7, and 11) and corresponding performance ceilings:

The 5-neuron seed plateaued at +14.1 reward.
The 7- and 11-neuron seeds reached expert performance (+18.4 and +18.7).
This suggests the "usable dimensionality" of the specific random projection sets a ceiling on achievable performance for a fixed task and capacity.

3. Necessity and Sufficiency of the Active Set

Ablation studies confirm that the sparse active set carries all task-relevant information:

FC1-Remove: Removing the active neurons in frozen agents crashes performance to random play (reward $\approx$ -21.0) across all seeds and frameworks.
FC1-Keep: Keeping only the active neurons (zeroing the rest) preserves the agent's full competence.
This holds true even when the policy is evaluated out-of-distribution (e.g., deterministic-trained agents evaluated with sticky actions), where the policy may fail but the representation remains necessary.

4. Early Commitment of the Bottleneck

A longitudinal sweep across 47 checkpoints (5M to $\sim$ 1.6B steps) reveals the temporal dynamics of bottleneck formation:

Early Lock: The active set is locked by 15–30M steps. The necessity of these neurons (FC1-Remove $\to$ random play) is detectable as early as 5M steps, before any reward improvement.
Lag: Reward turns positive 35–105M steps after the active set locks. The lag widens as the bottleneck capacity tightens (e.g., the 5-neuron seed takes $\sim$ 105M steps to learn to use its compressed representation, while the 11-neuron seed takes $\sim$ 35M).
Trainable Agents: Trainable agents also exhibit early commitment to an active set, but the content is qualitatively different: they commit to a broad set of neurons (43–64 of 64) rather than a narrow, sparse selection.

5. Structural Differences in Bottleneck Signatures (Breakout)

In Breakout, frozen and trainable agents reach competitive rewards via structurally different bottlenecks:

Frozen Agents: Use 17–25 active neurons with a high participation ratio ( $\sim$ 10–14), indicating a narrow set of moderately diverse, independent features.
Trainable Agents: Use 51 active neurons with a low participation ratio ( $\sim$ 3.6), indicating a broad set of highly correlated, redundant features.
This highlights that active count and participation ratio are complementary indicators of representational compression.

Significance and Claims

The paper claims to demonstrate that sparsity is not merely an engineered property but can emerge spontaneously from standard gradient descent when the feature extractor is fixed. The frozen random CNN acts as a "compressive measurement device" that reveals the intrinsic dimensionality of the underlying problem without explicit sparsity machinery.

Key implications drawn by the authors include:

Task Complexity Probing: A frozen-CNN training run can reveal, without explicit search, the approximate number of independent state variables an agent's policy utilizes, informing curriculum design and pruning targets.
Efficiency: Frozen CNNs enable extreme-efficiency deployment where only the trained readout (e.g., $\sim$ 9,400 parameters for Pong) needs to be stored, with the random seed sufficient to regenerate the projection.
Theoretical Connection: The work connects lottery ticket sparsity, reservoir computing, and compressed sensing. It suggests that gradient descent on a frozen random projection performs a sparse readout similar to how compressed sensing recovers signals, but driven by RL dynamics rather than explicit $\ell_1$ regularization.
Broader Applicability: The authors conjecture (without proving) that this mechanism could serve as a minimal-feature-set discovery tool in domains where input dimensionality dwarfs intrinsic task dimensionality (e.g., biological measurements) and as a lightweight diagnostic for adaptation-rank selection in fine-tuning (analogous to LoRA).

The authors remain modest regarding the generality of these findings, noting that the results are currently limited to discrete-action Atari games and the Nature-DQN architecture, and that a formal mathematical proof for why frozen random projections guarantee sparse solutions remains an open question.