🤖 machine learning

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

This paper introduces FRAME, a parameter-efficient fine-tuning method that employs a Mixture of Experts with learnable fractional-Fourier orders to dynamically adapt updates across a spatial-spectral continuum, thereby achieving superior performance and reduced interference compared to existing LoRA and spectral baselines.

Original authors: Tom Saliencro, Maya Lindqvist, Rohan Desai, Priya Nair, Daniel Whitmore

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tom Saliencro, Maya Lindqvist, Rohan Desai, Priya Nair, Daniel Whitmore

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, frozen library of knowledge (a Large Language Model) that you want to teach a new skill, like writing code or solving math problems. You don't want to rewrite the whole library because that's too expensive and slow. Instead, you want to add a small "sticky note" or a "cheat sheet" (a technique called PEFT or Parameter-Efficient Fine-Tuning) that teaches the model just what it needs to know.

For a long time, researchers had two main ways to write these cheat sheets:

The "Spatial" Way (LoRA): Writing the notes in the language of the original text (like standard English).
The "Spectral" Way (Fourier): Writing the notes in the language of frequencies and patterns (like musical notes or radio waves).

The problem? Sometimes the "Spatial" way is best, and sometimes the "Spectral" way is best. It depends on the task, the specific part of the model, and even the specific word being processed. But previous methods forced you to pick one way for the whole job.

Enter FRAME: The "Tunable Lens" Adapter

The authors introduce a new method called FRAME (Fractional-Fourier Mixture of Experts). Here is how it works, using simple analogies:

1. The "Zoom Lens" Analogy

Imagine the "Spatial" way is looking at a painting with your naked eye (seeing every brushstroke). The "Spectral" way is looking at the painting through a prism that breaks it down into pure colors (seeing the frequency of the light).

The authors realized there is a whole spectrum of lenses in between. You can have a lens that is slightly tilted, or half-way between the eye and the prism. This is called the Fractional-Fourier Transform.

FRAME gives every "expert" (a small helper module) its own tunable dial.
One expert might turn the dial to 0 (naked eye).
Another might turn it to 1 (prism).
A third might turn it to 0.4 (a blurry, half-way view).

The model learns exactly where to set the dial for every single task and every single word.

2. The "Specialized Team" Analogy

Think of FRAME as a team of specialists working on a project.

In older methods, everyone on the team was forced to speak the same language (e.g., everyone speaks English).
In FRAME, the team is a Mixture of Experts. When a new word comes in, a "router" (like a project manager) looks at it and asks: "Who is best suited to handle this?"
- If the word is a simple fact, the router might send it to the "Spatial" expert.
- If the word is part of a complex mathematical pattern, the router might send it to the "Spectral" expert.
- If it's something in between, it goes to an expert with a "mid-way" setting.

Because each expert is looking at the data through a different "lens," they don't get in each other's way. It's like having a team where one person looks at the big picture, another looks at the fine details, and a third looks at the rhythm of the text. They are decorrelated, meaning they don't repeat the same work or confuse each other.

3. The "Magic Shortcut"

You might think, "Wow, calculating all these different lenses must be super slow and expensive."
The authors found a clever mathematical shortcut (using something called a chirp-FFT). It's like having a magic calculator that can instantly switch between these different views without actually doing the heavy lifting.

Result: FRAME is almost as fast as the old methods, but much smarter.

What Did They Find?

The paper tested this on two big AI models (LLaMA and Qwen) across four types of tasks:

Commonsense (answering tricky questions)
Math (solving problems)
Code (writing software)
Knowledge (fact retrieval)

The Results:

Better Performance: FRAME beat all the previous best methods, including the "Spatial" ones and the "Spectral" ones.
Smarter Specialization: The model actually learned to set the dials differently for different tasks. For example, early layers of the model preferred one setting, while later layers preferred another.
Efficiency: It achieved these results while using fewer active parameters (less "computing power" per word) than many competitors.

The Bottom Line

The paper argues that the "domain" (the way we represent data) shouldn't be a fixed rule. It should be a learnable choice. FRAME proves that by giving AI models a dial to adjust how they "see" data—ranging from raw text to pure patterns—they can learn faster, make fewer mistakes, and handle complex tasks better than if they were forced to stick to just one way of thinking.

Technical Summary: FRAME (Fractional-Fourier Mixture of Experts)

1. Problem Statement

Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), typically reparameterize weight updates within a fixed basis. The dominant approach operates in the spatial domain (canonical coordinates), while a growing line of spectral methods operates in a fixed Fourier domain. The paper argues that this dichotomy represents an unexamined design choice: no single basis is optimal across all tasks, layers, or tokens.

Existing methods commit to either the spatial domain ( $a=0$ ) or the Fourier domain ( $a=1$ ), potentially wasting capacity if the optimal update representation lies in an intermediate domain. Furthermore, fixed-domain adapters often suffer from interference when merging or performing multi-task learning because updates may occupy overlapping subspaces. The core problem is that the adaptation domain itself should be a learnable degree of freedom rather than a fixed convention.

2. Methodology: FRAME

The authors propose FRAME (Fractional-Fourier Mixture of Experts), a PEFT adapter that treats the adaptation domain as a learnable, per-expert quantity.

Core Mechanism

FRAME utilizes the Fractional-Fourier Transform (FrFT), a one-parameter family of unitary transforms that continuously interpolates between the spatial domain and the Fourier domain.

The Parameter: The transform is defined by an order $a \in [0, 1]$ $a \in [0, 1]$ .
- $a=0$ : Identity transform (Spatial domain, equivalent to standard LoRA).
- $a=1$ : Standard Fourier transform (Spectral domain).
- $0 < a < 1$ : Intermediate domains representing a rotation in the time-frequency plane.
Architecture: FRAME is a Mixture-of-Experts (MoE) adapter. Each expert $i$ $i$ possesses:
1. Standard low-rank factors $A_i \in \mathbb{R}^{r \times d}$ and $B_i \in \mathbb{R}^{d_{out} \times r}$ .
2. A learnable scalar order $a_i \in [0, 1]$ .
Forward Pass: For a token $x$ , the update is computed as:
$\Delta W_i x = B_i A_i \text{Re}\{\Phi_{a_i} x\}$
where $\Phi_{a_i}$ is the discrete FrFT of order $a_i$ . The token is first rotated into the expert's specific fractional domain, the real part is taken, and the low-rank projection is applied.
Routing: A router selects the top- $k$ experts for each token, allowing different tokens to be adapted in different domains simultaneously.

Efficiency and Optimization

Computational Cost: Materializing the FrFT matrix explicitly would cost $O(d^2)$ . FRAME employs a chirp–FFT surrogate (Ozaktas et al., 1996) that computes the transform in $O(d \log d)$ time via pre-chirp multiplication, an FFT, and post-chirp multiplication. This adds negligible overhead compared to standard MoE-LoRA.
Optimization Strategy:
- Separate Optimizers: The order scalars $\{a_i\}$ and the matrix factors $\{A_i, B_i\}$ are trained with separate optimizers. The learning rate for orders ( $\eta_a$ ) is significantly smaller than for matrices ( $\eta$ ) because small changes in $a$ rotate the entire domain, whereas matrix updates are dense.
- Grouped Load Balancing: To prevent experts from collapsing to a single domain, the authors partition experts into $G$ "order bands" (initialized on a uniform grid). A balancing loss is applied within each band to ensure diverse domain utilization without forcing global uniformity.

3. Key Contributions

Reframing the Domain: The paper reframes the spatial-vs-spectral dichotomy as a single continuous axis (the fractional order) and demonstrates that the optimal adaptation domain varies by task, layer, and token.
FRAME Architecture: Introduction of a MoE adapter with per-expert learnable orders, an $O(d \log d)$ transform surrogate, and a specific optimization regime for the domain parameters.
Theoretical Guarantees:
- Strict Generalization: FRAME strictly generalizes both spatial LoRA and Fourier-domain adapters.
- Decorrelation: Theoretical proof that experts with well-separated orders are mutually incoherent (decorrelated), reducing interference between tasks and experts.
- Stability: Proofs that order gradients are bounded, ensuring stable training.
Empirical Performance: FRAME outperforms strong spatial and spectral baselines (including FlyLoRA, Fourier-MoE, and HMoRA) across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and QWEN2.5-7B, while maintaining a low active parameter budget.

4. Experimental Results

Performance: On LLaMA-3.1-8B, FRAME achieved an average commonsense accuracy of 87.1%, outperforming the strongest baseline (MoA, 86.3%) by +0.8% and vanilla LoRA by +2.6%. Similar gains were observed on QWEN2.5-7B.
Efficiency: FRAME activates only 0.31% of parameters (comparable to or lower than many MoE baselines) while achieving the best accuracy.
Ablation Studies:
- Fixing the order to $a=0$ (spatial) or $a=1$ (spectral) resulted in significant performance drops (−1.4% and −1.1% respectively), confirming that a single fixed domain is insufficient.
- Using a fixed diverse grid of orders (non-learned) recovered only part of the performance gap, highlighting the necessity of learning the specific order per expert.
- The chirp–FFT surrogate incurred less than a 7% training time overhead compared to spatial MoE-LoRA, whereas a dense transform implementation was 3× slower.
Mechanism Analysis:
- Specialization: Learned orders specialized by task and layer depth (shifting from near-spatial in early layers to more spectral in later layers).
- Decorrelation: Inter-expert coherence decreased monotonically as the difference in their orders increased, validating the theoretical decorrelation property.
- Token Routing: Different token types (e.g., numerals vs. function words) were routed to systematically different orders.

5. Significance and Claims

The paper claims that the adaptation domain is a learnable design choice, not a fixed convention. By introducing a continuous spectrum of domains via the fractional-Fourier order, FRAME allows the model to place low-rank updates in the domain where they are most compact and least interfering.

The authors emphasize three specific advantages:

Expressivity: A mixture spanning multiple orders represents updates that no single-domain adapter of the same rank can capture.
Decorrelation: Different orders induce mutually incoherent subspaces, naturally reducing interference between experts and tasks without requiring hard orthogonality constraints.
Efficiency: The learnable domain adds negligible computational cost due to the efficient chirp–FFT surrogate.

The paper concludes that FRAME provides a "broadly useful axis" for adaptation, suggesting that the fractional lens could be applied to full fine-tuning, attention mechanisms, or continual learning, though these are left as future work. The authors note limitations regarding the non-zero overhead of the transform and the sensitivity to routing collapse, which they mitigate but do not fully eliminate.