🤖 machine learning

Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

This paper introduces an online, off-policy reinforcement learning framework that utilizes Hadamard overparametrization to derive sparse, interpretable Gaussian-mixture-model Q-functions, enabling efficient Riemannian optimization that achieves superior parameter efficiency and generalization compared to deep RL methods.

Original authors: Minh Vu, Konstantinos Slavakis

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Minh Vu, Konstantinos Slavakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a robot doesn't just follow a rigid instruction manual but learns by doing, much like a child learning to ride a bike. This is the realm of Reinforcement Learning (RL), a branch of artificial intelligence where an agent interacts with its environment, tries different actions, and learns from the rewards or punishments it receives. The goal is to figure out the best possible strategy to maximize happiness (or points) over time. To do this, the agent needs a "map" of how good every possible move is in every situation. In the world of AI, this map is called a Q-function.

For a long time, making these maps was tricky. If the world is simple, you can just write down every single possibility in a giant list. But if the world is complex and continuous—like a drone flying through a forest or a car driving on a highway—the list becomes impossibly huge. So, scientists started using "black box" tools called Deep Neural Networks to guess the map. These are powerful, but they are also heavy, hungry for computer power, and hard to understand; you can't easily see why the network decided a certain move was good. On the other hand, there are simpler, more transparent models, but they often struggle to keep up with the speed and chaos of real-time learning. The big question is: Can we build a learning agent that is fast, lightweight, and easy to understand, without sacrificing its smarts?

This paper introduces a clever new way to build these learning maps, called Sparse Gaussian-Mixture-Model Q-Functions (S-GMM-QFs). Think of the agent's mind as a collection of "experts," where each expert is a simple, bell-shaped curve (a Gaussian) that knows how to handle a specific part of the world. Instead of forcing the agent to pick a fixed number of experts beforehand, the authors give it a massive pool of 500 potential experts and a special "magic eraser" called Hadamard overparametrization. As the agent learns, this magic eraser automatically wipes out the experts that aren't useful, leaving behind only the few that truly matter.

The result is a model that starts with a huge, flexible brain but quickly prunes itself down to a lean, efficient one. The authors tested this on video-game-style challenges like landing a spacecraft on the moon and flying a bird through pipes. They found that this new method learns just as fast, or even faster, than the heavy "black box" deep learning models, but it uses a tiny fraction of the computer power. Best of all, because the remaining experts are simple shapes with clear positions and sizes, we can actually look at the model and see exactly where it is focusing its attention. It's like swapping a mysterious, unexplainable supercomputer for a team of specialized, transparent guides that you can actually talk to.

The Core Idea: A Garden of Experts

To understand how this works, let's imagine the agent is trying to learn a new video game. In the past, scientists tried two main approaches. The first was to use a giant, dense neural network—a "black box" with millions of connections. It's like hiring a massive, anonymous army of soldiers to solve the problem. It works, but it's expensive, slow to update, and you have no idea which soldier is actually doing the heavy lifting. The second approach was to use a simple model with a fixed, small number of parts. This is like hiring a tiny, specific team. It's fast and cheap, but if the team is too small, they might miss crucial details of the game.

The authors of this paper decided to try a third path: Start big, then get smart.

They created a model that begins with a huge pool of 500 "Gaussian experts." Each expert is a simple mathematical shape (a bell curve) that represents a specific region of the game world. For example, one expert might know how to handle the "left side of the screen," while another knows how to handle "falling fast." Initially, the model has all 500 experts active, ready to help.

Here is where the magic happens. The authors used a technique called Hadamard overparametrization. In plain English, instead of giving each expert a single "importance score," they broke that score down into a product of three smaller numbers. Imagine each expert's importance is the result of three different judges voting. If any one of those judges votes "zero," the expert's total importance becomes zero.

As the agent learns from its experiences (like landing on the moon or crashing into a pipe), it adjusts these judges' votes. The "magic" is that this setup naturally pushes the votes of useless experts toward zero. It's like a garden where you plant 500 seeds. As the seasons change (the agent learns), the plants that aren't suited for the soil naturally wither away, leaving only the strongest, most relevant ones. The model doesn't need a human to come in and manually cut the weak branches; the learning process itself does the pruning.

Why This Matters: Speed, Smarts, and Clarity

The paper tested this idea on two famous challenges: the Lunar Lander (landing a spaceship) and Flappy Bird (navigating pipes). They compared their new "pruning" method against the heavy-duty Deep Neural Networks (like DQN and PPO) that are currently the standard in the field.

The results were surprising and promising. In these simulations, the new method matched or even beat the performance of the massive deep learning models. But the real win was in efficiency. While the deep learning models required a massive amount of computer calculations (measured in FLOPs) to learn, the new method learned just as well using significantly fewer resources. In the Flappy Bird game, for instance, the new method reached high scores much faster than the deep learning models, which seemed to struggle with the delayed rewards of the game.

Perhaps the most exciting part is interpretability. With a deep neural network, if the agent makes a mistake, it's hard to know why. It's a black box. But with this new method, because the model prunes itself down to a few specific "experts," we can look at the remaining ones and see exactly what they are doing. The paper shows that the surviving experts align perfectly with the important parts of the game. For the Lunar Lander, the experts cluster around the landing pad and the obstacles. It's as if the agent is saying, "I'm focusing on these specific areas because that's where the action is." This transparency is something deep learning models simply can't offer without complex, extra tools.

The Catch and the Future

Of course, no magic is perfect. The authors are careful to note that this method works best when the "world" isn't too huge. If the state space (the number of things the agent needs to track) gets too big—like if the agent had to process raw images from a camera—the math gets too heavy because of the way the model handles shapes and curves. The paper suggests that for now, this is best for problems with moderate complexity, not for raw, high-definition video feeds.

However, the authors are already looking ahead. They suggest that this approach could be combined with other techniques to handle those bigger, messier worlds. They also note that while their current tests were on games with discrete actions (like "jump" or "don't jump"), the math could be adapted for continuous actions (like "steer slightly left").

In the end, this paper offers a refreshing alternative to the "bigger is better" mindset in AI. It suggests that by starting with a large, flexible pool of simple ideas and letting the learning process naturally select the best ones, we can build agents that are not only powerful and efficient but also transparent and understandable. It's a reminder that sometimes, the smartest thing an AI can do is know what not to think about.

Technical Summary: Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

Problem Statement

Reinforcement learning (RL) faces significant challenges in online, off-policy settings, particularly regarding the trade-off between model expressiveness, computational efficiency, and interpretability. Traditional tabular methods fail in continuous state-action spaces, while nonparametric kernel-based approaches suffer from unbounded model growth and high memory overhead as data accumulates. Conversely, Deep Reinforcement Learning (Deep RL) methods, such as Deep Q-Networks (DQNs), offer strong representational power but rely on large "black-box" architectures with millions of parameters. These deep models are computationally expensive to update in real-time, struggle with distributional shifts in non-stationary environments, and lack intrinsic interpretability. Furthermore, existing sparsification techniques for deep networks (e.g., pruning or sparse training) often fail to establish explicit links between surviving connections and the geometric structure of the input space, limiting their transparency.

The paper addresses the need for a lightweight, interpretable, and scalable Q-function approximator that can adapt to streaming, non-stationary data while maintaining geometric consistency and generalization in low-parameter regimes.

Methodology

The proposed framework introduces Sparse Gaussian-Mixture-Model Q-Functions (S-GMM-QFs) within an online, off-policy policy-iteration (PI) scheme. The methodology integrates three core components:

GMM-QF Formulation:
Instead of modeling the probability density of Q-values (as in distributional RL), the Q-function is directly approximated as a weighted sum of multivariate Gaussian components:
$Q(s, a) = \sum_{k=1}^K \xi_k(a) G(s | m_k, C_k)$
where $\xi_k(a)$ are learnable weights, and $m_k, C_k$ are the mean and covariance of the $k$ -th Gaussian component. The parameter space is a Cartesian product of Euclidean spaces and the manifold of symmetric positive definite (SPD) matrices, inheriting a natural Riemannian manifold structure.
Hadamard Overparametrization for Sparsification:
To avoid the non-smoothness of $\ell_p$ -norms (which are incompatible with Riemannian optimization) and the difficulty of pre-specifying the number of components $K$ , the authors employ Hadamard overparametrization. The weight matrix $\Xi$ is factorized into $J$ auxiliary matrices $\Upsilon_j$ such that $\Xi = \Upsilon_1 \odot \Upsilon_2 \odot \dots \odot \Upsilon_J$ .
- Smooth Regularization: A smooth Frobenius norm penalty is applied to the factors $\Upsilon_j$ . This induces an implicit bias toward sparsity in the product $\Xi$ without requiring non-differentiable thresholds.
- Adaptive Pruning: The model starts with a large pool of components ( $K$ ) and adaptively identifies meaningful ones. Uninformative components are driven to negligible weights, effectively pruning the model during training.
Riemannian Optimization and Experience Replay:
- Optimization: The policy evaluation step minimizes a Bellman-residual (BR) loss using Riemannian gradient descent (specifically a Riemannian extension of Adam, or RAdam). This respects the geometric constraints of the SPD manifold for covariance updates via retraction mappings and parallel transport for momentum.
- Experience Buffer: To handle off-policy learning and distributional mismatch, an experience buffer is utilized. The paper proposes and evaluates two novel sampling strategies to replace standard Importance Sampling (IS) weighting, which is structurally incompatible with Riemannian updates:
  - Frequency-based Priority Decay: A mechanism that penalizes over-sampled transitions to flatten the distribution without distorting the manifold geometry.
  - Online Clustering: A diversity-based strategy that enforces equal sampling quotas across clusters of state-action transitions, ensuring broad coverage of the feature space.

Key Contributions

The paper outlines four primary contributions:

(C1) Online Off-Policy Framework: Extends GMM-QFs from offline, on-policy settings to an online, off-policy PI framework. This reconciles streaming, non-stationary data with the Riemannian structure of the parameter space and actively mitigates distributional mismatch via a dedicated experience buffer.
(C2) Sparse GMM-QFs via Hadamard Overparametrization: Introduces S-GMM-QFs, which enable interpretable sparsification through smooth regularization. Unlike deep RL sparsification, this approach allows components to evolve based on task structure, yielding sparse models where each surviving component's geometric role (mean and covariance) is explicitly encoded.
Riemannian Optimization: Detailed the computation of Riemannian gradients and updates (including parallel transport and retraction) for the specific manifold structure of GMM-QFs, enabling stable online learning.
Novel Replay Strategies: Proposed frequency-based priority decay and online clustering buffers to address the incompatibility of standard IS weighting with Riemannian optimization, enhancing both exploration and sample efficiency.

Results

Numerical experiments were conducted on two continuous control benchmarks: Lunar Lander and Flappy Bird.

Performance vs. Deep RL: S-GMM-QFs matched or exceeded the performance of Deep RL baselines (DQN, PPO) and their sparsified variants (Pruned, SET, RigL). Notably, S-GMM-QFs achieved faster improvement per observed transition.
Parameter Efficiency: In low-parameter regimes where sparsified Deep RL methods degraded significantly, S-GMM-QFs maintained strong generalization. The sparse models achieved peak performance with substantially fewer parameters than required by dense Deep RL models to reach similar plateaus.
Computational Efficiency: When measured against cumulative Floating-Point Operations (FLOPs), S-GMM-QFs demonstrated superior efficiency. The sparsification process reduced the computational cost of Riemannian retractions and gradient calculations during training, allowing the model to reach optimal performance with orders of magnitude fewer FLOPs than dense counterparts.
Interpretability: Visualizations of the learned Gaussian components revealed that S-GMM-QFs concentrate on a small subset of components that align with the state visitation density of the optimal policy. Unlike deep networks, where sparsity is opaque, the surviving Gaussian components in S-GMM-QFs provide a transparent, geometric interpretation of the Q-function's structure without requiring post-hoc attribution tools.
Buffer Strategies: The proposed "fair decay" and "clustering" buffers outperformed standard uniform and prioritized experience replay (PER) in specific contexts. Clustering was particularly effective for the delayed-reward Flappy Bird task, while fair decay performed well on the dense-reward Lunar Lander.

Significance and Claims

The paper claims that the proposed framework offers a principled alternative to Deep RL for online settings, particularly where interpretability and computational constraints are critical. By leveraging the geometric structure of the parameter space and smooth overparametrization, the method achieves a "sweet spot" where parameter efficiency and interpretability combine to maintain strong generalization in regimes where sparsified deep RL approaches fail.

The authors emphasize that S-GMM-QFs are not merely a model-compression technique applied post-training but a dynamic learning mechanism that converts representational redundancy into performance early in training. The geometric self-documentation of the model—where surviving components explicitly map to regions of the state space—provides a level of transparency unattainable by black-box neural networks.

The paper acknowledges limitations, noting that the cubic computational complexity of Riemannian operations ( $O(D_s^3)$ ) currently restricts the method to moderate state dimensions, making it unsuitable for raw high-dimensional sensory inputs (e.g., images) without state representation learning. Future work is directed toward actor-critic extensions for continuous actions and online state representation learning.