🤖 AI

EUGens: Efficient, Unified, and General Dense Layers

This paper introduces EUGens, a new class of efficient, unified, and general dense layers that leverage random features and input norms to approximate fully-connected layers with linear inference complexity and reduced parameters, while enabling unbiased approximation of polynomial activations and efficient adaptation to pre-trained models, resulting in significant improvements in inference speed and memory efficiency across various tasks.

Original authors: Sang Min Kim, Byeongchan Kim, Arijit Sehanobish, Somnath Basu Roy Chowdhury, Rahul Kidambi, Dongseok Shim, Avinava Dubey, Snigdha Chaturvedi, Min-hwan Oh, Krzysztof Choromanski

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sang Min Kim, Byeongchan Kim, Arijit Sehanobish, Somnath Basu Roy Chowdhury, Rahul Kidambi, Dongseok Shim, Avinava Dubey, Snigdha Chaturvedi, Min-hwan Oh, Krzysztof Choromanski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-end restaurant (a neural network) that serves millions of customers every day. The most expensive and time-consuming part of your operation isn't the cooking itself, but the menu planning.

In modern AI, this "menu planning" is done by Fully Connected Feedforward Layers (FFLs). These are the layers where the AI takes a huge amount of information, multiplies it by a massive list of rules (weights), and produces a result. The problem is that as the restaurant grows, this menu planning becomes incredibly slow and requires a huge amount of storage space. It's like trying to calculate the perfect meal for every single customer by writing out a unique, 10,000-page recipe book for each one.

The paper introduces a new kitchen tool called EUGens (Efficient, Unified, and General dense layers) to solve this. Here is how it works, using simple analogies:

1. The "Magic Filter" vs. The "Heavy Calculator"

The Old Way (Standard FFLs):
Imagine you have a giant calculator. To get an answer, you have to press every single button on the keypad for every single customer. If you have 1,000 customers, you press the buttons 1,000 times. This is quadratic complexity—it gets slow very fast.

The New Way (EUGens):
EUGens act like a magic filter. Instead of pressing every button, the filter takes the customer's order (the input) and the chef's rules (the weights) and transforms them into a much shorter, simpler list of ingredients.

The Trick: It uses something called Random Features. Think of this as a special, pre-made spice blend. Instead of measuring every single spice from scratch for every dish, the chef uses this blend to approximate the flavor.
The Result: The calculation changes from "pressing every button" to just "mixing a few bowls." This turns a slow, quadratic process into a fast, linear one. If you have 1,000 customers, you only do a fraction of the work.

2. The "Shape Shifter" (Unifying Different Methods)

Before this paper, scientists had different "hacks" to make AI faster, but they were like different tools for different jobs: one tool worked for math problems, another for pictures, and another for text.

EUGens are like a Swiss Army Knife. They are "Unified." Whether you are teaching a robot to see (Computer Vision), a computer to speak (Language Models), or a system to build 3D worlds (NeRFs), EUGens can replace the heavy menu planning in all of them with the same efficient tool.

3. The "Magic Mirror" (Unbiased Approximation)

One of the biggest fears in simplifying things is losing accuracy. If you use a shortcut, will the food taste wrong?

The paper claims EUGens are unbiased. Imagine a magic mirror that reflects a complex, high-definition image. Even though the mirror is made of simple, low-resolution tiles, the reflection is so accurate that you can't tell the difference.
The authors prove mathematically that EUGens can mimic the complex "flavors" (activation functions like ReLU or GELU) of the original heavy layers without needing to retrain the whole system from scratch.

4. The "Instant Upgrade" (Knowledge Distillation)

Usually, if you want to upgrade a restaurant's kitchen, you have to close it down, fire the staff, and retrain everyone from day one. This takes months.

The paper introduces a distillation technique that acts like a "copy-paste" upgrade. You can take an existing, pre-trained AI model (a fully trained restaurant) and swap out the heavy menu planners for EUGens instantly.
Because of the math behind EUGens, you don't need to retrain the whole thing. You just calculate the new settings using a simple formula (no back-and-forth guessing). This allows you to speed up existing models immediately.

What Did They Actually Achieve?

The paper tested this "magic filter" in three specific areas:

Language Models (LLMs): They replaced the heavy layers in a model like GPT-2. Result: The model became 27% faster at thinking and used 30% less memory, while still writing good text.
Image Classification (Vision): They tested it on models that identify objects in photos (like ImageNet). Result: The models stayed just as accurate but ran much faster.
3D Scene Reconstruction (NeRFs): They used it to build 3D worlds from 2D photos. Result: Rendering these 3D worlds became 24% to 27% faster, making it possible to create realistic scenes in real-time.

Summary

Think of EUGens as a way to replace a heavy, slow, custom-built engine in a race car with a lightweight, high-tech engine that runs just as fast but uses half the fuel. It doesn't change the destination (the AI still learns the same things); it just gets you there much quicker and with less baggage.

Technical Summary: EUGens: Efficient, Unified, and General Dense Layers

Problem Statement

Modern machine learning architectures, particularly Transformers (used in Large Language Models and Vision Transformers) and Implicit Neural Representations (INRs) like Neural Radiance Fields (NeRF), rely heavily on Fully-Connected Feedforward Layers (FFLs). While central to model expressivity, FFLs introduce significant computational and parameter bottlenecks. Their standard implementation involves quadratic scaling in time and space complexity relative to the hidden layer size, hindering the deployment of large-scale models in real-time and resource-constrained environments. Existing solutions, such as dimensionality reduction (Johnson-Lindenstrauss Transform), pruning, quantization, or specific kernel methods, often lack generality, are limited to specific activation functions, or fail to provide unbiased approximations for arbitrary polynomial activations.

Methodology: EUGens

The authors propose EUGens (Efficient, Unified, and General dense layers), a new class of dense layers that generalize standard FFLs. EUGens leverage Random Features (RF) to approximate FFL computations while introducing a direct dependence on input norms.

Core Mechanism

Unlike standard FFLs where the output is $f(Wx)$, EUGens disentangle the weight matrix $W$ and the input vector $x$ . They transform both into low-dimensional representations via non-linear mappings $f$ and $g$ , such that the layer output is computed as a dot product:
$\text{EUGen}(W, x) = g(W)^\top f(x)$

The transformations involve:

Input/Weight Augmentation: Concatenating the input $x$ with $1$ and $\|x\|^2$ (and similarly for weights).
Random Projections: Applying a series of random matrices $G^i_j$ to these augmented vectors.
Non-linear Mapping: Applying element-wise functions $\Psi$ and $\Phi$ to the results of Hadamard products of these projections.
Order $k$ : The "order" of the layer controls the expressiveness, allowing the approximation of polynomial activation functions of degree $k$ .

Theoretical Foundations

Unbiased Approximation: The paper proves (Theorem 3.1) that EUGens provide an unbiased estimation of FFLs with arbitrary polynomial activation functions. This is achieved by sampling entries of the projection matrices from specific zero-mean distributions.
Generalization to Continuous Functions: By leveraging the Bernstein Theorem, the authors extend these results to show that EUGens can approximate FFLs with general continuous activation functions (Theorem 3.4), as any continuous function on a closed interval can be approximated by polynomials.
Concentration Bounds: Theoretical analysis (Theorems 3.2 and 3.3) provides concentration results, demonstrating that the variance of the estimation decreases with the number of random features $m$ , and the probability of large deviation from the mean is exponentially small.
Complexity Reduction: By decoupling weights and inputs, inference complexity is reduced from quadratic $O(d^2 + dl)$ to linear $O(mdk^2 + ml)$ (or $O(mdk + ml)$ with specific optimizations), where $m \ll d$ .

Practical Enhancements

Quasi-Monte Carlo (QMC): The authors incorporate structured random matrices (e.g., Gaussian Orthogonal Matrices) to further reduce variance in the estimation.
Analytic Distillation: A layer-wise knowledge transfer technique is proposed that bypasses backpropagation. By storing inputs/outputs of a pre-trained layer, the EUGen parameters can be optimized analytically (closed-form solution) to mimic the original layer, enabling "zero-shot" integration into pre-trained models.

Key Contributions

Proposal of EUGens: A new dense layer class that unifies existing efficient extensions and generalizes FFLs using random features and input norm dependence.
Theoretical Guarantees: The first unbiased algorithms approximating FFLs with arbitrary polynomial activation functions, accompanied by rigorous concentration bounds.
Empirical Superiority: Demonstrated that EUGens accurately approximate standard activations (ReLU, GeLU, Softplus) with lower Mean Squared Error (MSE) than baselines like Low-Rank and SNNK.
Broad Integration: Successful integration into diverse architectures including Transformers (LLMs, ViTs) and INRs (NeRF, iSDF, Mip-NeRF 360).
Distillation Framework: A method to replace FFLs in pre-trained models without retraining, achieving speedups without the overhead of full model reconstruction.

Experimental Results

The authors evaluated EUGens across multiple tasks, reporting significant gains in inference speed and memory efficiency:

Language Modeling (GPT-2): In pre-training on OpenWebText, EUGens achieved validation losses comparable to vanilla GPT-2 while using significantly fewer parameters. Replacing FFLs with EUGens allowed for substantial parameter reduction with minimal loss in performance.
Image Classification (ViT): On ImageNet and Places365, EUGen-based ViTs matched the accuracy of vanilla ViTs while reducing inference parameters by nearly 30% (e.g., replacing 6 layers).
3D Scene Reconstruction (NeRF & iSDF):
- NeRF: Achieved a 24% reduction in inference time and 30% reduction in model size with virtually no loss in reconstruction quality (PSNR).
- Mip-NeRF 360: Achieved a 27% increase in inference speed.
- iSDF: Achieved 22.6% faster inference and 5% faster training compared to standard MLP-based iSDF.
Distillation: The analytic distillation approach successfully recovered the quality of original NeRF models while improving inference speeds by up to 26%, all without backpropagation.

Significance and Claims

The paper claims that EUGens represent a significant step toward the scalable deployment of large-scale neural networks in real-world scenarios. By reducing the computational complexity of FFLs from quadratic to linear time and significantly lowering parameter counts, EUGens enable faster inference and lower memory usage without sacrificing the expressive power of the original models.

The authors emphasize that EUGens are orthogonal to other efficiency techniques (like pruning or sparse sampling) and can be combined with them. Furthermore, the ability to distill EUGens into pre-trained models without retraining addresses a major barrier to adopting efficient layers in existing production systems. The work highlights that efficient neural networks are essential for applications in robotics, computer vision, and NLP, where real-time performance and resource constraints are critical.