🤖 AI

FRACTAL: SSM with Fractional Recurrent Architecture for Computational Temporal Analysis of Long Sequences

The paper introduces FRACTAL, a novel state space model that integrates fractional measure theory into recursive memory updates to resolve the trade-off between retaining unbounded history and detecting short-term variations, achieving superior performance on long sequence benchmarks compared to existing models like S5.

Original authors: Mengqi Li, Wensheng Lin, Jinshuai Yang, Lixin Li

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mengqi Li, Wensheng Lin, Jinshuai Yang, Lixin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to remember a long story. You face a tricky problem: if you try to remember every single detail equally well, the most recent words get drowned out by the thousands of words that came before them. But if you focus only on the last few sentences to catch the immediate plot twist, you forget the beginning of the story entirely.

For a long time, computer models trying to learn from long sequences (like text, stock prices, or heartbeats) had to choose between these two bad options. This paper introduces a new model called FRACTAL that solves this problem by using a mathematical trick called "fractional calculus."

Here is a simple breakdown of how it works, using everyday analogies:

1. The Problem: The "Memory Trilemma"

Think of memory like a bucket of water.

Old Model A (Uniform): It pours water in at a steady rate. The bucket is full, but the water at the top (the newest stuff) gets diluted by the water at the bottom (the old stuff). You remember everything, but the newest details are faint.
Old Model B (Exponential): It pours water in, but the bottom of the bucket has a hole. The old water leaks out quickly, so the newest water stays fresh and bright. But if the story is too long, the bottom of the bucket is empty, and you lose the context of the beginning.
Old Model C (Window): It only keeps the last few cups of water. You have perfect clarity on the immediate past, but you have zero memory of anything that happened before that.

The paper argues that real-world data (like financial markets or biological signals) needs all three: you need the full history, you need to hear the newest details clearly, and you need to handle events that happen at different speeds.

2. The Solution: The "Fractional" Magic

FRACTAL introduces a new way to fill the bucket using a tunable singularity.

Imagine the bucket has a special shape that changes based on a dial you can turn (called the singularity index, or $\alpha$ ).

The Shape: Instead of a straight cylinder or a cone, the bucket has a "heavy tail." It holds onto the old water (history) very well, but the very top of the bucket is shaped like a funnel that concentrates the newest water right at the surface.
The Dial:
- If you turn the dial one way, the bucket acts like a standard memory (remembering everything equally).
- If you turn it the other way, it acts like a "recent memory" (focusing on the now).
- The Magic: FRACTAL can sit anywhere in between. It creates a "power-law" memory, which means it remembers the distant past just enough to keep the context, but it gives a massive boost to the very recent past so you don't miss sudden changes.

3. The Architecture: The "Spectral Filter Bank"

The paper doesn't just use one bucket; it uses a whole orchestra of buckets (called a Multi-Scale Filter Bank).

Imagine a choir where every singer has a slightly different voice.
Some singers are tuned to remember the slow, long-term trends (like the overall mood of a song).
Other singers are tuned to catch the fast, sudden notes (like a drumbeat or a sudden shout).
FRACTAL assigns different "dials" to different parts of its brain. Some parts focus on the long history, while others zoom in on the immediate present.
The model then mixes these voices together to understand the whole picture.

4. Why It's Better (The Results)

The authors tested FRACTAL on a standard set of difficult puzzles called the Long Range Arena. These puzzles require the computer to find connections between things that are very far apart in a sequence.

The Score: FRACTAL scored 87.11%, beating the previous best model (S5) which scored 87.04%.
The Big Win: On a specific task called ListOps (which involves solving nested math problems), FRACTAL scored 61.85%, beating the previous best of 61.10%. This task is hard because you need to remember the opening bracket from the very beginning of the sentence while processing the numbers at the end. FRACTAL's ability to balance "long history" with "recent focus" made it better at this.

Summary

FRACTAL is a new type of computer brain that stops forcing a choice between "remembering everything" and "remembering the now." By using a mathematical tool called fractional calculus, it creates a memory that naturally holds onto the distant past while staying hyper-aware of the present moment. It's like having a library where the books are organized so you can instantly grab the latest bestseller without losing track of the classics.

Technical Summary: FRACTAL – State Space Model with Fractional Recurrent Architecture

1. Problem Statement

Sequence modeling in continuous time faces a fundamental tension: the need to retain an unbounded historical context while simultaneously detecting high-resolution, abrupt short-term variations. Existing State Space Models (SSMs), particularly those relying on the High-Order Polynomial Projection Operators (HiPPO) framework, are constrained by a "trilemma" of memory measures:

Scaled Legendre (LegS): Assigns uniform weight to history, ensuring timescale invariance but progressively diluting the influence of recent inputs.
Translated Laguerre (LagT): Prioritizes recency via exponential decay but imposes a fixed characteristic timescale, sacrificing robustness to time-warped signals and global context.
Translated Legendre (LegT): Offers high local resolution via a sliding window but catastrophically discards context beyond that fixed horizon.

No existing measure simultaneously satisfies full-history retention, recency sensitivity, and scale invariance. This limitation hinders the application of SSMs in physical and biological domains characterized by multi-scale dynamics, such as financial volatility, physiological $1/f$ noise, and network traffic, which exhibit power-law decay kernels rather than exponential or uniform distributions.

2. Methodology: The Fractional HiPPO Framework

The paper proposes FRACTAL (Fractional Recurrent Architecture for Computational Temporal Analysis of Long sequences), which integrates fractional calculus into the HiPPO framework to resolve the memory trilemma.

2.1 Fractional-Order Measure

The core theoretical innovation is the introduction of a fractional-order measure $\mu(t)(x)$ defined by a tunable singularity index $\alpha \in [0, 1)$ :
$\mu(t)(x) = \frac{1-\alpha}{t^{1-\alpha}} (t-x)^{-\alpha} \mathbb{I}_{[0,t]}(x)$
This measure interpolates between the uniform distribution (LegS, where $\alpha=0$ ) and a concentrated mass at the present. It induces a power-law weighting $(t-\tau)^{-\alpha}$ , creating a heavy tail for long-term retention while enforcing a singularity at the current timestep to amplify local sensitivity. Crucially, this power-law structure preserves scale invariance under temporal dilation, unlike exponential measures.

2.2 Theoretical Derivation

Orthogonal Basis: The optimal projection for this measure is derived using Jacobi polynomials $P_n^{(-\alpha, 0)}$ , which generalize the Legendre polynomials used in standard HiPPO-LegS.
State Space Dynamics: By applying a dimensionless transformation and integration by parts to the projection coefficients, the authors derive a Linear Time-Varying (LTV) system:
$\frac{d}{dt}x(t) = -\frac{1}{t} A(\alpha) x(t) + \frac{1}{t} B(\alpha) u(t)$
The $1/t$ factor is the mathematical signature of scale invariance.
System Matrices:
- Input Projection ( $B$ ): A closed-form solution is derived where the binomial coefficient decays slower than in LegS, effectively amplifying high-frequency input components.
- State Transition ( $A$ ): The matrix $A(\alpha)$ is lower triangular. A key theoretical result is that its diagonal elements are invariant to $\alpha$ ( $A_{nn} = n+1$ ), ensuring the eigenvalues remain fixed at $\{1, 2, \dots, N\}$ . This guarantees spectral stability regardless of the memory concentration profile.

2.3 The FRACTAL Architecture

To enable efficient parallel training, the LTV system is relaxed to a Linear Time-Invariant (LTI) system by replacing the $1/t$ factor with a learnable timescale parameter $\Delta$ .

Spectral Initialization: Unlike prior works that approximate matrix structures, FRACTAL performs a direct numerical eigendecomposition of $A(\alpha)$ (which is strictly diagonalizable) to obtain the diagonal spectrum $\Lambda$ .
Physically-Informed Initialization: The input projection vector $\tilde{B}$ is initialized using the analytic closed-form solution derived from the fractional measure, acting as a preconditioner to align initial dynamics with the power-law memory structure.
Multi-Scale Filter Bank: The state space is partitioned into channels, each assigned a distinct singularity index $\alpha_k$ $α_{k}$ . This creates a "Fractional Filter Bank" where:
- Low $\alpha$ channels act as low-pass filters (retaining global context).
- High $\alpha$ channels act as high-pass filters (amplifying local transients).
Computation: The diagonalized system is solved via parallel prefix-sum (associative scan) algorithms, maintaining $O(N \log L)$ complexity.

3. Key Contributions

Theoretical: The introduction of a fractional-order power-law measure into the HiPPO framework. This is the first measure to theoretically satisfy the "impossible trinity" of full-history retention, recency sensitivity, and scale invariance. The authors provide rigorous derivations showing that Jacobi polynomials form the natural basis and that the state transition matrix's eigenvalues are invariant to the singularity index.
Architectural: The instantiation of this theory into the FRACTAL architecture, featuring a multi-channel configuration with heterogeneous memory parameters (a spectral filter bank) and an analytic initialization strategy that leverages the closed-form solution for the input projection.
Empirical: Comprehensive validation on the Long Range Arena (LRA) benchmark, demonstrating that the fractional measure yields empirical gains in multi-scale reasoning tasks.

4. Experimental Results

Evaluated on the Long Range Arena (LRA) benchmark, FRACTAL achieves an average score of 87.11%, outperforming the S5 model (87.04%) and S4 (86.09%) under strictly identical single-GPU conditions.

ListOps: FRACTAL achieves 61.85%, surpassing S5 (61.10%) and S4 (59.60%). This task requires parsing nested mathematical expressions, validating the model's ability to simultaneously track global bracket structures (low $\alpha$ ) and local operand values (high $\alpha$ ).
Path-X: On sequences of length 16K, FRACTAL achieves 98.39%, effectively matching S5 (98.62%), confirming that the fractional singularity does not compromise gradient propagation over extreme horizons.
Numerical Verification: Experiments confirm that the memory density shifts from uniform to heavy-tailed as $\alpha$ increases, and that the diagonal elements of the state transition matrix remain invariant, as predicted by theory.

5. Significance and Claims

The paper claims that FRACTAL bridges the gap between the optimal projection theory of HiPPO and the memory properties of fractional calculus. By generalizing the measure-theoretic foundation of SSMs, the work provides a rigorous mechanism to tune memory kernels between pure history and pure recency without sacrificing scale invariance.

The authors position FRACTAL as a foundational innovation within the Linear Time-Invariant (LTI) SSM paradigm. They explicitly distinguish their work from "Selective" (time-varying) SSMs like Mamba, arguing that comparing a foundational, train-from-scratch LTI innovation against heavily optimized, pre-trained selective architectures would conflate distinct research paradigms. Consequently, the paper asserts that the fractional measure offers a structural advantage specifically in the power-law regime, making it particularly suitable for domains characterized by multi-scale dynamics and heavy-tailed temporal dependencies, such as financial time series and physiological signal analysis.