🤖 machine learning

Composing Linear Layers from Irreducibles

This paper proposes a differentiable algorithm using Clifford algebra to decompose linear layers into compositions of bivectors and rotors, achieving a parameter-efficient $O(\log^2 d)$ representation that matches the performance of dense matrices and other approximations in LLM attention mechanisms.

Original authors: Travis Pence, Daisuke Yamada, Vikas Singh

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Travis Pence, Daisuke Yamada, Vikas Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, complex machine (like a modern AI) that is built out of millions of tiny, individual gears. Usually, to make this machine work, engineers have to design a unique, custom gear for every single connection. This makes the machine huge, heavy, and expensive to run because it has to carry all those millions of gears with it.

This paper asks a simple question: Do we really need millions of unique gears? Or can we build the whole machine using just a few basic, reusable shapes?

Here is the breakdown of their discovery, using everyday analogies:

1. The Problem: The "Heavy Suitcase"

Current AI models are like people trying to carry a suitcase filled with millions of different, custom-made tools. Even if they only need to do a simple task, they have to drag all those tools along. This makes the AI slow and requires a lot of memory (storage space) to hold all the "weights" (the numbers that define how the gears turn).

2. The Solution: The "Lego Set" of Rotors

The authors discovered that instead of using millions of custom tools, you can build the same complex functions using a small set of geometric primitives called rotors.

The Analogy: Think of a standard AI layer as a giant, solid block of clay that you have to sculpt from scratch every time. The authors' method is like having a small box of Lego bricks (specifically, "bivectors," which are like flat, oriented planes).
How it works: Instead of sculpting a new shape, you just snap a few of these Lego bricks together in a specific order. By combining these simple "rotating" pieces, you can recreate the exact same complex movement that the giant clay block used to do.

3. The Magic Trick: "Sandwiching"

The paper uses a mathematical tool called Clifford Algebra (which is like a super-charged version of the geometry we learn in school).

The Metaphor: Imagine you have a piece of paper (your data). To change its shape, you don't need to redraw the whole thing. Instead, you take two special "rotating frames" (rotors) and put the paper between them. You rotate the paper from the left, then from the right.
The Result: This "sandwich" action rotates and transforms the data perfectly. The amazing part is that you only need a handful of these frames to do the job of millions of standard numbers.

4. The Results: Same Performance, Tiny Footprint

The researchers tested this idea by swapping out the "heavy gears" in the brains of some popular AI models (specifically the parts that help the AI understand language).

The Comparison: They replaced the standard, heavy layers with their new "Lego rotor" layers.
The Outcome: The AI models performed just as well as before. They could still answer questions and predict the next word in a sentence with the same accuracy.
The Win: However, the new models were drastically smaller.
- A standard layer might need 4 million parameters (numbers) to work.
- The new rotor layer needed only about 1,000 parameters.
- That is a reduction of roughly 4,700 times.

5. Why This Matters (According to the Paper)

The paper doesn't claim this will immediately cure diseases or solve world hunger. Instead, it focuses on the efficiency of the machine itself:

Less Memory: Because the model is so much smaller, you don't need a massive computer to run it. It's like switching from a semi-truck to a compact car.
Speed Potential: While the current software version is still being optimized, the authors suggest that because the data is so much smaller, it could eventually run much faster on standard hardware, especially because it reduces the need to constantly load huge amounts of data from memory.
Understanding: It proves that complex AI behavior doesn't necessarily require complex, messy math. It can be built from a clean, structured set of geometric building blocks.

Summary

The paper shows that we can rebuild the "brain" of an AI using a tiny, efficient set of geometric building blocks (rotors) instead of millions of custom numbers. It's like realizing you can build a skyscraper using a few types of standard bricks stacked in the right way, rather than needing a unique, custom stone for every single window and door. The building stands just as tall and strong, but it's much lighter and easier to carry.

Technical Summary: Composing Linear Layers from Irreducibles

Problem Statement
Contemporary large language models (LLMs) exhibit behaviors suggesting they rely on low-level primitives that compose into modules with richer functionality. However, the fundamental building blocks of these linear transformations remain poorly understood. While prior work has explored parameter-efficient approximations (e.g., Low-Rank Adaptation, Block-Hadamard) and modular compositions (e.g., Mixture-of-Experts), there is a lack of understanding regarding the mechanistic composition of low-level primitives that generate complex linear behaviors. The authors pose a concrete problem: Can the functionality of a linear layer, which typically contains millions of parameters ( $O(d^2)$ ), be synthesized from a minimal set of irreducible geometric objects?

Methodology
The authors propose an algebraic framework using Clifford algebra to decompose and reconstruct linear transformations.

Algebraic Formulation:
- Standard linear maps are traditionally represented as dense matrices. The authors instead express linear transformations as compositions of bivectors (geometric objects encoding oriented planes) within the Clifford algebra $Cl(n)$.
- They utilize the Spin group $Spin(n)$, a double cover of the special orthogonal group $SO(n)$. Elements of this group, called rotors ( $r$ ), act on multivectors via the "sandwich product" $x \mapsto rxr^\dagger$ .
- While arbitrary multivectors require $2^n$ parameters, the authors restrict the construction to rotors generated by bivectors. A rotor $r$ is the exponential of a bivector $b$ : $r = \exp(b)$ .
Parameter Efficiency:
- A dense $d \times d$ matrix requires $O(d^2)$ parameters.
- By representing the transformation as a composition of rotors acting on subspaces, the number of required scalar parameters scales as $O(\log_2 d)$ . Specifically, for a width $w$ , the parameter count is approximately $w \binom{\log_2 d}{2}$ . For $d=2048$ , this reduces millions of parameters to a few hundred.
Differentiable Decomposition Algorithm:
- The exponential map for general bivectors involves an infinite series, which is computationally problematic. However, for simple bivectors (those representing a single plane of rotation), the exponential map has a closed-form solution: $\exp(b) = \cos(\|b\|) + \sin(\|b\|)\frac{b}{\|b\|}$ .
- To handle general bivectors, the authors employ an invariant decomposition (based on Roelfs and Keninck [2021]), which expresses any bivector as a sum of at most $k = \lfloor n/2 \rfloor$ mutually commuting, orthogonal simple bivectors.
- They introduce a differentiable invariant decomposition algorithm (Algorithms 1 and 2) that iteratively extracts these simple bivectors using a Krylov subspace-inspired power iteration method adapted for Clifford algebra. This avoids explicit eigendecomposition, ensuring numerical stability and differentiability for gradient-based optimization.
Generalized Rotor Gadget:
- To handle arbitrary input/output dimensions ( $d_{in}, d_{out}$ ) that are not powers of 2, the authors construct a generalized transformation $\psi$ by aggregating multiple rotor-sandwich modules operating on different subspaces, followed by a pooling operator.

Key Contributions

Algebraic Representation: The authors express linear transformations as compositions of geometric primitives (bivectors) using rotor sandwich products, reducing parameter complexity from $O(d^2)$ to $O(\log_2 d)$ .
Differentiable Decomposition: They propose a closed-form, differentiable algorithm to map arbitrary bivectors to their corresponding rotors via invariant decomposition, enabling integration with autograd frameworks.
Empirical Validation: The method is applied to the Key, Query, and Value projections in LLM attention layers. The rotor-based layers are trained to mimic dense layers (layer-wise) and trained end-to-end on FMNIST.

Results

Parameter Efficiency: In experiments with LLaMa-3.2 1B and Qwen-2.5 1.5B, rotor layers achieved massive parameter reductions. For instance, the query projection in LLaMa-3.2 1B dropped from ~4.19M parameters (dense) to $\le 896$ parameters (rotor), a ~4700x reduction.
Performance: Despite the extreme compression, rotor-based layers matched or outperformed strong baselines, including Low-Rank (LR1, LR4) and Block-Hadamard (BH1) approximations.
- On Wikitext2 (LLaMa-3.2 1B), replacing a single attention layer yielded a log-PPL of 2.629 for rotors, compared to 2.636 for the best baseline (BH1).
- On Arc Challenge, rotors achieved 55.31% accuracy (LLaMa-3.2 1B), outperforming BH1 (54.83%) and LR4 (53.84%).
Robustness: The performance remained consistent across various layer combinations (replacing 1, 2, or 3 layers) and different datasets (Wikitext2, C4, PTB, Arc, HellaSwag).
End-to-End Training: When trained from scratch on FMNIST, the rotor-based MLP converged to a final accuracy (88.36%) comparable to the dense baseline (89.67%), though convergence was slightly slower.

Significance and Claims
The paper claims to demonstrate the feasibility of synthesizing linear layers from a compact set of geometric building blocks. The authors emphasize that their goal is not merely to provide a drop-in replacement for dense layers (which currently suffers from higher wall-clock inference times due to lack of hardware-optimized kernels) but to provide an algebraic perspective on how low-level primitives compose into higher-level functions.

The findings suggest that the complex functionality of linear layers in deep models can be realized through a structured composition of a few hundred geometric objects (rotors) rather than millions of independent weights. This offers a new avenue for:

Interpretability: Understanding linear layers as compositions of oriented planes.
Parameter Efficiency: Drastically reducing the parameter count while maintaining performance.
Future Hardware: Potential to trade compute for memory bandwidth by synthesizing weights on-chip from geometric parameters, addressing the memory bandwidth bottlenecks in large model inference.

The authors remain modest, noting that practical benefits require additional system-level integration and hardware-aware implementations beyond the scope of this work.