🤖 machine learning

HalfNet: Randomized Neural Networks with Learned Subspace Geometry

The paper introduces HalfNet, a neural network approach that learns a low-rank factorization of the covariance matrix to generate data-dependent random weights, achieving performance comparable to fully trained models with significantly fewer parameters by leveraging the geometry of the weight space.

Original authors: Ethem Alpaydin

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ethem Alpaydin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize pictures of cats and dogs. Usually, to do this, you have to teach the robot every single detail from scratch. You have to show it millions of examples and adjust millions of tiny knobs (called "weights") one by one until it gets it right. This takes a lot of time, memory, and computing power.

The paper introduces a new method called HalfNet. Instead of teaching the robot every single knob, HalfNet teaches the robot a rulebook for how to set those knobs randomly, but in a smart way.

Here is how it works, broken down with simple analogies:

1. The "Random Guess" vs. The "Smart Random"

Traditionally, some researchers tried just setting the knobs to random numbers and seeing if the robot could learn the rest. It's like giving a student a test where the questions are random, but the student can study the answers. It works okay, but not great.

HalfNet says: "Let's keep the questions random, but let's teach the student how the questions are related to each other."

The Analogy: Imagine you are throwing darts at a board.
- Standard Random: You throw darts completely blindfolded, hitting anywhere on the board with equal chance.
- HalfNet: You are still blindfolded, but you are told, "Hey, the board is tilted. Most darts will land in this specific oval shape." You don't know exactly where each dart will land, but you know the shape of where they are likely to go. HalfNet learns that "oval shape" (the geometry) from the data.

2. The "Low-Rank" Shortcut

The paper explains that even in complex neural networks, the "important" information usually lives in a smaller, simpler space.

The Analogy: Think of a high-resolution photo of a sunset. It has millions of pixels. But if you squint your eyes, you realize the whole picture is mostly just a gradient of orange and purple. You don't need to remember the exact color of every single pixel to understand it's a sunset.
HalfNet's Trick: It realizes that the "knobs" in the robot's brain don't need to be independent. They can be grouped together. Instead of learning 1,000 separate rules, HalfNet learns just 10 or 20 "master rules" (called the rank) that describe how the 1,000 knobs should behave together.

3. How It Saves Space

Because HalfNet only needs to learn these "master rules" (the shape of the distribution) rather than every single knob, it uses way fewer parameters.

The Result: The paper tested this on two famous image datasets (MNIST with handwritten numbers and CIFAR-10 with colorful toys).
- On MNIST, HalfNet achieved the same accuracy as a fully trained, heavy-duty robot, but with half the number of settings to adjust.
- On CIFAR-10, it matched the performance of a massive robot while using 8 times fewer parameters.

4. What Does This Actually Mean?

The authors did a "spectral analysis" (basically, they looked at the math behind the scenes) and found that most of the "brainpower" of a trained neural network is actually just about the geometry (the shape and relationships) of the weights, not the precise value of every single number.

The Metaphor: It's like building a house. A standard builder cuts every single brick to a specific size. HalfNet says, "Let's just build a mold that makes bricks in the right shape and pattern. We don't need to measure every brick individually; the mold does the work."

5. Other Cool Features

The paper also shows that this idea is flexible:

Binary Weights: You can even make the robot use only "Yes/No" (1 or -1) switches instead of complex numbers, which saves even more memory, and HalfNet still works well.
Layers: You can stack these "smart random" layers on top of each other, or put them after standard image filters (convolutions).

Summary

HalfNet is a new way to build AI that says: "Don't memorize every detail. Instead, learn the pattern of how the details should be arranged."

By learning the "shape" of the randomness rather than the randomness itself, the AI becomes much more efficient. It gets just as good at recognizing images as the heavy, slow models, but it does it with a fraction of the memory and computing power. The paper concludes that for tasks where data has strong patterns (like images), learning this "geometry" is the secret sauce to efficiency.

Technical Summary: HALFNET: RANDOMIZED NEURAL NETWORKS WITH LEARNED SUBSPACE GEOMETRY

1. Problem Statement

Traditional neural network training involves optimizing every weight parameter individually, which can be computationally expensive and parameter-heavy. Conversely, randomized neural networks (e.g., Extreme Learning Machines) fix weights to values drawn from a standard distribution (e.g., $N(0, I)$ ), simplifying optimization but often sacrificing predictive power because the random geometry may not align with the data structure.

The core problem addressed is how to retain the simplicity of randomized networks while capturing the data-dependent structure necessary for high accuracy. Specifically, the paper investigates whether the geometry of the weight distribution (the covariance structure) is more critical for predictive performance than the precise values of individual weights, and if this geometry can be learned efficiently.

2. Methodology: HalfNet

The proposed HalfNet framework introduces a stochastic parameterization where weights are not fixed to a standard distribution nor learned individually, but are sampled from a learned distribution $N(0, \Sigma)$ .

2.1. Model Formulation

For a hidden unit $h$ with input $x$ , the activation is defined as:
$y_h = g(u_h(r_h^\top x) + u_{h0})$
where $g$ is the activation function (e.g., ReLU), and $u_h, u_{h0}$ are learned scaling and translation parameters specific to the unit.

The random projection vector $r_h$ is generated via a low-rank factorization:
$r_h = B z_h + \text{diag}(\sigma) \epsilon_h$
where:

$z_h \sim N_k(0, I)$ and $\epsilon_h \sim N_d(0, I)$ are random vectors sampled once and fixed.
$B \in \mathbb{R}^{d \times k}$ (with $k < d$ ) and $\sigma \in \mathbb{R}^d$ are learned parameters shared across all units in the layer.
The resulting distribution is $r_h \sim N_d(0, \Sigma)$ , where $\Sigma = BB^\top + \text{diag}(\sigma^2)$ .

This approach decouples the randomness (the vectors $z_h, \epsilon_h$ ) from the geometry (the matrix $B$ and scaling $\sigma$ ). The rank $k$ acts as a hyperparameter controlling model complexity, interpolating between fully random units ( $k \to 0$ ) and fully trainable networks.

2.2. Extensions

The framework is extended to:

Binary Weights: By applying a sign function to the generated vectors, HalfNet can produce binary ( $\pm 1$ ) weights. The sign operation preserves angular relationships defined by the learned covariance $\Sigma$ , allowing the model to learn dependencies even in the binary regime.
Convolutional Layers: Filters are generated using the same low-rank factorization, assuming a shared covariance structure across filters within a layer.
Multi-layer Architectures: Multiple HalfNet layers can be stacked, with separate learned parameters for each layer.

3. Key Contributions

Learning Covariance Structure: HalfNet learns the covariance structure of random projections rather than individual weights, effectively performing a data-dependent random embedding.
Smooth Scaling with Rank: The paper demonstrates that accuracy scales smoothly with the rank $k$ of the covariance factor, suggesting that much of the predictive power in trained layers resides in low-dimensional geometric structures.
Versatility: The framework is shown to be applicable to fully-connected layers, convolutional front-ends, and binary weight networks.
Theoretical Interpretations: The method admits three complementary interpretations:
- Random Projection: Learning a linear embedding $x' = B^\top x$ followed by fixed random projections.
- Metric Learning: Learning a Mahalanobis metric where $B$ determines which input directions are emphasized.
- Kernel Methods: Learning the distribution of projection vectors implicitly learns the kernel function.

4. Experimental Results

4.1. MNIST Experiments

Single Hidden Layer (784–64–10): HalfNet variants with learned $\sigma$ and low-rank $B$ significantly outperform purely random networks. With rank $k=4$ , HalfNet matches the accuracy of a Linear Perceptron (LP) with nearly half the parameters. With $k=16$ , it matches a fully trained Multilayer Perceptron (MLP) with comparable parameter counts.
Spectral Analysis: Singular Value Decomposition (SVD) of weights from a fully trained MLP reveals that 60% of spectral energy lies in the top 9 directions and 80% in the top 17. This confirms that trained weights lie in a low-dimensional subspace, which HalfNet explicitly parameterizes.
Convolutional Frontend: Adding a convolutional layer (16 filters) improves all models. HalfNet benefits significantly, achieving high accuracy even at low ranks ( $k=1$ ), as the complexity depends on the effective rank rather than the increased fan-in.
Two Hidden Layers: Stacking HalfNet layers (784–64–64–10) maintains the trend of improved accuracy with increasing rank, proving the method's scalability within deep networks.
Binary Weights: While binarization at very low ranks ( $k=2$ ) causes accuracy drops due to collapsed feature diversity, $k \ge 4$ yields strong results. At $k=16$ , binary HalfNet achieves 94.87% accuracy, close to the 95.30% of its continuous counterpart.

4.2. CIFAR-10 Experiments

Architecture: A simple network with a single convolutional layer (32 filters) followed by a large fully-connected layer (256 units). The FC layer contains >99% of the parameters.
Performance: The "Full–Half" configuration (trainable Conv, HalfNet FC) matches the fully trained "Full–Full" baseline with approximately 8x fewer parameters at rank $k_f=32$ . At $k_f=48$ , it slightly exceeds the baseline performance.
Convolutional HalfNet: Applying HalfNet to the convolutional layer ("Half–Half") initially decreased performance and slowed training, suggesting that the low-rank assumption may be too restrictive for layers that already possess strong structural constraints (locality, weight sharing).

5. Significance and Claims

The paper claims that HalfNet demonstrates that learning the geometry of projection distributions can capture the majority of the predictive power of fully trained neural layers while using substantially fewer parameters.

Key takeaways emphasized by the author include:

Geometry over Values: The predictive power of neural networks lies more in the geometry of the weight space (covariance structure) than in the precise values of individual parameters.
Efficiency: HalfNet offers a smooth trade-off between fixed random features and fully trainable networks, allowing practitioners to tune model capacity via the rank parameter $k$ .
Distinction from Low-Rank Factorization: Unlike deterministic low-rank methods (e.g., $W=UV$ where both are learned), HalfNet learns the distribution from which weights are sampled, keeping the projection vectors stochastic. This connects the method to random features and kernel learning.
Limitations: The benefits are most pronounced in high fan-in settings with strong data correlations (e.g., fully-connected layers on image data). The paper notes that performance gains may be reduced in settings with weaker correlations, and further evaluation on diverse datasets is needed.

The work positions HalfNet not merely as an architectural trick for compression, but as a fundamental shift toward learning the underlying metric and kernel structure of the data through randomized embeddings.