🤖 machine learning

Hadamard Representation: Scaffolding Performance Across Model-free RL

The paper proposes the Hadamard Representation, a simple architectural modification that replaces standard hidden layers with element-wise products of two independent layers to prevent neuron dormancy and increase effective rank, thereby consistently improving the performance of diverse model-free reinforcement learning algorithms across multiple domains without requiring hyperparameter tuning.

Original authors: Jacob E. Kooi, Zhao Yang, Mark Hoogendoorn, Vincent François-Lavet

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jacob E. Kooi, Zhao Yang, Mark Hoogendoorn, Vincent François-Lavet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a robot to play video games or walk like a human. You give it a "brain" made of a deep neural network, which is essentially a massive team of tiny workers (neurons) passing information along.

The paper argues that as these robots learn, their brains start to get sick. Specifically, many of the workers stop working entirely, and the ones that keep working start to get stuck in a loop, repeating the same useless thought over and over. This makes the robot's brain less capable, even though it has been training for a long time.

The authors propose a simple fix called the Hadamard Representation (HR). Think of it as a clever architectural upgrade that stops the brain from getting sick and helps it think more creatively.

Here is a breakdown of the problem and the solution using everyday analogies:

The Problem: The "Silent Saboteurs"

In the world of AI, there are two main types of "workers" (activation functions) used to process information: ReLU and Tanh.

The ReLU Worker (The "Dead" Employee):
Imagine a worker who, when they get tired or confused, simply stops talking and outputs a "0." If they output zero, the next person in the chain ignores them completely. It's like a silent employee who has been fired; they are gone, but they don't cause any trouble. The team just works around them.
The Tanh Worker (The "Stuck" Employee):
This is the tricky one. When a Tanh worker gets tired, they don't stop talking. Instead, they get stuck shouting either "YES!" (+1) or "NO!" (-1) forever, no matter what the situation is.
- The Danger: Because they are still shouting, the next person in the chain listens to them. But since the shout is always the same, it acts like a hidden, unchangeable bias. It's like a stuck radio playing the same song in the background that distorts the conversation. The robot thinks this constant noise is part of the real world, which messes up its decisions. The paper calls this "silently corrupting" the network.

The Solution: The "Hadamard Representation" (HR)

The authors suggest a simple fix: instead of having one worker do the job, have two independent workers do the job, and then make them multiply their answers together.

Imagine a committee making a decision.

Old Way: One person speaks. If they are stuck shouting "YES," the whole committee is biased toward "YES."
New Way (HR): Two people speak independently. The final decision is only made if both of them agree on a specific combination of their thoughts.

This creates two powerful benefits:

1. The "Safety Net" Effect (Stopping the Stuck Shout)

For the "stuck" Tanh workers to become useless, both of the independent workers would have to get stuck at the exact same time.

Analogy: Imagine two people trying to get stuck in a mud pit. It's hard for one person to get stuck. It is extremely unlikely that two people, standing in different spots, will get stuck in the exact same way at the exact same time.
Result: The "stuck" workers become much rarer. The brain stays flexible and doesn't get corrupted by those hidden, constant biases.

2. The "Double Vision" Effect (Richer Thinking)

Even if the workers aren't stuck, having two people look at the problem and multiply their insights creates a much richer conversation.

Analogy: If you ask one person to describe a cat, they might say "furry." If you ask two people and multiply their descriptions, you might get a complex understanding like "furry AND fast."
Result: The robot can understand more complex patterns without needing to hire more workers (add more neurons). It gets smarter without getting bigger.

What They Found

The researchers tested this idea on three very different types of challenges:

Playing Atari Games: (Like Pong or Breakout). Here, the "stuck worker" problem was huge. Using HR made the robots play significantly better, beating the old methods by a wide margin.
Walking Robots (State-based): Robots that learn to walk using sensors (like a robot dog). Here, the "stuck worker" problem was rare, but the "Double Vision" effect still helped the robots learn faster and walk better.
Walking Robots (Visual-based): Robots that learn to walk just by looking at a camera screen. Again, HR improved performance without needing any extra tuning.

The Bottom Line

The paper shows that deep learning agents often lose their ability to learn because their internal "workers" get stuck in bad habits. By simply changing the architecture to use two parallel paths that multiply their results, the authors created a system that:

Prevents workers from getting stuck in a loop.
Allows the brain to understand more complex ideas without getting bigger.
Works across many different types of robots and games without needing to tweak the settings.

It's a small structural change that acts like a vaccine against the "brain rot" that happens during long training sessions.

Technical Summary: Hadamard Representation: Scaffolding Performance Across Model-free RL

Problem Statement

Deep reinforcement learning (RL) agents suffer from a progressive loss of representational capacity during training. This manifests in two primary ways: neurons becoming dormant (losing plasticity) and the collapse of effective rank (redundancy among surviving neurons). While existing remedies like periodic resets or specialized architectures exist, they are often algorithm- or domain-specific.

A critical, under-analyzed failure mode involves smooth activation functions, specifically tanh. Unlike the "dying ReLU" phenomenon where dormant neurons output zero and are effectively pruned, saturated tanh neurons collapse to non-zero constants ( $\pm 1$ ). The authors demonstrate that these saturated neurons do not merely stop learning; they actively corrupt downstream layers by converting their outgoing weights into fixed, input-independent biases. This "silent" distortion degrades performance even when the count of dormant neurons is lower than in ReLU networks.

Methodology: Hadamard Representation (HR)

The authors propose a simple, drop-in architectural modification called the Hadamard Representation (HR). Instead of a standard hidden layer $z(x) = f(Ax + b)$, HR replaces it with the element-wise product of two independently parameterized branches:

$z(x) = f(A_1x + b_1) \odot f(A_2x + b_2)$

where $f$ is the activation function (e.g., tanh or ReLU), and $(A_1, b_1)$ and $(A_2, b_2)$ are independent sets of weights and biases.

HR operates through two complementary mechanisms:

Saturation Resistance (Mechanism 1):
- For tanh, a neuron becomes dormant (saturated) only if both independent branches saturate simultaneously. Assuming independence, if the probability of a single branch saturating is $p$ , the probability of collapse drops to $p^2$ . This significantly reduces the incidence of dormant tanh neurons.
- Conversely, for ReLU, saturation (outputting zero) occurs if either branch saturates. The probability of collapse increases to $2p - p^2$ . Thus, HR is beneficial for tanh but detrimental for ReLU in high-dormancy regimes.
- This mechanism prevents the "silent corruption" of downstream layers by ensuring gradients do not vanish as easily and neurons do not lock into fixed biases.
Multiplicative Expressiveness (Mechanism 2):
- Independently of dormancy, the multiplicative structure captures second-order interactions between two views of the input. This expands the function class at a fixed layer width, increasing the effective rank of the hidden representations.
- This mechanism provides benefits even in regimes where dormant neurons are already rare (e.g., in state-of-the-art continuous control methods using LayerNorm), acting as an independent source of representational gain.

Key Contributions

Identification of a Unique Failure Mode: The paper identifies that saturated tanh neurons are strictly more harmful than dormant ReLU neurons because they introduce fixed biases rather than being pruned.
Architectural Solution (HR): Introduction of a lightweight, symmetric multiplicative layer that reduces tanh collapse probability quadratically while increasing effective rank.
Mechanism Analysis: Theoretical and empirical separation of HR's benefits into "saturation resistance" (dominant in high-dormancy settings) and "multiplicative expressiveness" (dominant in low-dormancy settings).
Hyperparameter-Free Improvement: HR improves performance across diverse algorithms and domains without requiring any hyperparameter tuning.

Experimental Results

The authors evaluated HR across five algorithms and three distinct domains:

Pixel-based Discrete Action (Atari):
- DQN & PPO: On 8 Atari games, HR applied to tanh activations reduced dormant neuron fractions by ~23% and increased effective rank, leading to >10% performance gains over strong baselines.
- PQN: On a large-scale evaluation of 51 Atari games, HR with tanh improved median human-normalized scores by over 100% compared to standard tanh, and closed the performance gap with ReLU while maintaining higher effective rank.
- Comparison: HR outperformed other plasticity-preserving methods like CReLU and ReDo.
State-based Continuous Control (HumanoidBench):
- Using SimbaV2 (a state-of-the-art method with negligible dormant neuron rates), HR still improved performance by 18%. This confirms that Mechanism 2 (multiplicative expressiveness) drives gains even when dormancy is not the primary bottleneck.
Visual Continuous Control (DMC-Visual):
- Using MR.Q on visual tasks, HR improved performance by 17%.
- In extended training on difficult Humanoid tasks, MR.Q + HR matched the performance of the state-of-the-art XQC method, outperforming both the MR.Q baseline and DrQ-v2.

Ablation Studies:

Parameter Count: The authors compared HR against "widened" baselines with matched parameter counts. Widening the network often degraded performance (e.g., -46% in DQN) or provided minimal gains, whereas HR consistently improved results. This rules out parameter count as the sole explanation for the improvements.
Stacking: Stacking multiple HR blocks (2HR) provided no additional benefit over a single HR block, suggesting the first layer is sufficient to capture the representational gains.

Significance and Claims

The paper claims that the Hadamard Representation offers a universal, architecture-level fix for representational capacity loss in model-free RL. Its significance lies in:

Universality: It improves performance across value-based (DQN), policy-gradient (PPO), and vectorized (PQN) algorithms, as well as state-based and visual continuous control.
Simplicity: It requires no hyperparameter tuning and can be integrated into existing codebases as a drop-in layer replacement.
Theoretical Insight: It reframes the understanding of "dormant" neurons, highlighting that for smooth activations, dormancy is an active source of error (bias injection) rather than passive pruning.
Future Potential: The authors suggest that the potential of non-piecewise-linear activations in RL has been underestimated and that HR opens new avenues for architectural design in both RL and broader machine learning contexts.

The work concludes that while HR is a minimal change, it addresses a fundamental limitation in deep RL training dynamics, providing consistent gains where other methods fail or require specific domain adaptations.