🤖 machine learning

Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions

This paper introduces a synthetic benchmarking framework that utilizes controllable memory functions to systematically evaluate and compare the performance of various sequence modeling architectures across a continuum of temporal structures, revealing new insights into their approximation capabilities, optimization dynamics, and architectural trade-offs.

Original authors: Haotian Jiang, Zeyu Bao, Shida Wang, Qianxiao Li

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haotian Jiang, Zeyu Bao, Shida Wang, Qianxiao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to tell a story. To do this, the robot needs to understand not just the words it hears right now, but how those words connect to things it heard seconds, minutes, or even hours ago. This is the world of sequence modeling, a branch of computer science where machines learn to process data that flows over time, like speech, weather patterns, or the sentences in a novel. The big challenge for scientists is that different stories have different "rhythms." Some stories rely on a quick, fading memory (like remembering the last word you spoke), while others depend on a single, distant fact that matters deeply (like a plot twist mentioned in the first chapter that explains the ending).

For years, engineers have built different types of "robot brains" to handle these rhythms. Some, like Recurrent Neural Networks (RNNs), are like scribes who read a story one word at a time, trying to keep a running note in their head. Others, like Transformers, are like readers who can flip back and forth through the whole book instantly to find connections. But here's the problem: we don't really know exactly which brain is best for which kind of story. Real-world data is messy and confusing, making it hard to tell if a robot is failing because the story is too hard or because the robot's brain is the wrong shape. Scientists needed a way to test these robots with perfectly clean, custom-made stories where they could control exactly how the memory worked.

This paper introduces a clever new way to test these sequence models by creating "synthetic targets"—essentially, artificial memory games with mathematically perfect rules. The researchers invented a system using memory functions, which act like dials that control how the past influences the present. They created four distinct types of memory challenges:

Exponential and Polynomial Decay: Imagine a fading echo. The sound is loud at first but gets quieter and quieter. Some models are great at this, while others struggle when the echo fades very slowly.
Impulse: This is like a single, distant shout. The answer depends entirely on one specific word from way back in the past, with nothing in between.
Airy: This is a spiky, oscillating pattern, like a heartbeat that skips or a signal that is only active in specific, sparse moments.

The team put four famous robot brains—LSTM (a type of RNN), S4D (a state-space model), TCN (a convolutional model), and the Transformer—through these tests. They found that no single robot is a superhero for every situation. The LSTM and S4D models were excellent at handling the fading echoes (exponential decay) but completely collapsed when faced with the distant shout (impulse) or the slow, fading polynomial echo. Interestingly, making these models "deeper" (adding more layers) didn't always help; for the fading echoes, a simple single layer was best, but for the distant shouts, they needed at least three layers to succeed.

On the other hand, the TCN (convolutional model) was a champion at handling the distant shouts and sparse patterns, thanks to its ability to look far back in time efficiently. However, the researchers discovered that for the TCN, "sparsity" (how many zeros are in the memory) wasn't the only thing that mattered. They introduced a new concept called "tail energy complexity," which measures how much "weight" is left in the tail of the memory. They found that even if a pattern isn't perfectly sparse, if the tail of the memory has too much energy, the TCN struggles to learn it.

The Transformer model showed a fascinating trade-off. Its performance depended heavily on how its "attention heads" (the parts of the brain that focus on different things) were sized. If the memory was strong and complex, having too many small attention heads actually hurt performance; there was a "sweet spot" where the number of heads and their size had to be perfectly balanced.

Perhaps the most exciting discovery came when they tried mixing these brains together. They found that if you let a weaker model (like the LSTM) process the data first and then pass it to a stronger model (like the TCN or Transformer), the combined system could fix the weaknesses of the first one. It's like having a junior assistant who summarizes the story for a senior editor; the editor can then do a much better job than if they had to read the raw, messy notes themselves.

In short, the paper suggests that there is no "one-size-fits-all" architecture for sequence modeling. The best choice depends entirely on the specific "rhythm" of the memory you are trying to capture. By using these controllable, synthetic memory functions, researchers can now precisely diagnose which brain is failing and why, paving the way for smarter, more tailored AI systems that know exactly how to listen to the past.

Technical Summary: Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions

Problem Statement

The evolution of sequence modeling architectures—from Recurrent Neural Networks (RNNs) and Convolutional models to Transformers and Structured State-Space Models (SSMs)—has been driven by the need to capture diverse temporal dependencies. However, systematically characterizing the strengths and limitations of these architectures remains a fundamental challenge. Current evaluations rely heavily on large, real-world benchmarks (e.g., Long Range Arena, LongBench) where temporal structures are complex and undefined. This lack of control makes it difficult to isolate which specific temporal properties drive performance differences. While theoretical studies exist regarding how architectures handle dependencies (e.g., via impulse response analysis or approximation theory), a systematic, controlled methodology to directly test these properties across different models is lacking.

Methodology

The authors propose a synthetic benchmarking framework centered on parametric memory functions $\rho(s, \alpha)$ to generate controllable sequence targets.

1. Theoretical Foundation

The framework is grounded in the representation of linear, causal, time-homogeneous sequential targets as mappings between function spaces. Theorem 2.1 establishes that such targets admit a unique memory function representation:
$y(t) = \sum_{s=0}^{t} \rho(H)(s)x(t-s)$
where $\rho(H)$ uniquely determines the temporal dependency structure. The authors extend this to nonlinear targets by introducing activation functions $\sigma_1, \sigma_2$ , demonstrating that the memory function $\rho(s, \alpha)$ still dominates the temporal dependency structure (defined via partial derivatives) even in the presence of nonlinearity.

2. Parametric Memory Functions

Four representative memory functions are defined to probe distinct temporal structures, controlled by a parameter $\alpha \in [0, 1]$ representing temporal strength:

Exponential Decay ( $\rho_{exp}$ ): Models rapidly decaying dependencies.
Polynomial Decay ( $\rho_{poly}$ ): Models slowly decaying dependencies.
Impulse ( $\rho_{\delta}$ ): Models sparse, long-range dependencies where output depends on a single distant input.
Airy ( $\rho_{Ai}$ ): Models sparsity patterns with oscillatory behavior, where effective sparsity is controlled by shifting the function.

3. Experimental Setup

The framework evaluates four representative architectures: LSTM (recurrent), S4D (structured state-space), TCN (convolutional), and Transformers (attention-based).

Evaluation Metrics: The primary focus is on approximation error (estimated via final training loss on large synthetic datasets to prevent overfitting) and optimization dynamics (sensitivity to initialization and training speed).
Scaling: To ensure fair comparison, scaling functions $\mu(\alpha)$ are defined to normalize the total area or shift range of memory functions relative to sequence length $T$ . A relative Mean Squared Error (MSE) is used to normalize loss across different output scales.
Ablation: Experiments are conducted across varying model sizes, depths, and sequence lengths to ensure robustness.

Key Contributions and Results

1. Recurrent Architectures (LSTM, S4D)

Decay Sensitivity: These models perform well on exponential decay but struggle significantly with polynomial decay and sparse long-range dependencies (Impulse/Airy).
Depth Effects: Depth provides benefits for sparse dependencies (requiring 3+ layers to avoid degradation) but offers no advantage (and may harm optimization) for smoothly decaying structures (Exponential/Polynomial), where a single layer is sufficient.
Prony Series Analysis: The difficulty in approximating polynomial decay is attributed to the need for recurrent models to fit a Prony series with modes spread across many timescales, creating an ill-conditioned optimization landscape.

2. Temporal Convolutional Networks (TCN)

Long-Range Dependencies: TCNs maintain stable performance on Impulse targets regardless of dependency range, leveraging dilated convolutions to expand the receptive field.
Tail Energy Complexity: The authors introduce a novel tail energy complexity measure $C(\rho, s)$ , which captures the magnitude decay of memory functions beyond binary sparsity. This measure correlates strongly with TCN approximation error, explaining why performance degrades on Airy and polynomial targets as $\alpha$ increases, even when binary sparsity remains constant.
Optimization Sensitivity: TCNs exhibit high sensitivity to initialization and optimizer choice (Adam vs. AdamW), particularly on sparse and oscillatory targets.

3. Transformers

Effective Rank Theory: Transformer performance aligns with effective rank theory. Targets with higher effective ranks (Exponential, Polynomial, Airy as $\alpha$ increases) yield higher approximation errors, while low-rank Impulse targets remain stable.
Head-Width Trade-off: The optimal number of attention heads depends on the memory function type and temporal strength. For strong dependencies, a non-monotonic relationship exists where too many heads (reducing per-head dimension) can degrade performance, indicating a critical trade-off between head count and dimensionality.

4. Mixed Architectures and Targets

Sequential Composition: Combining architectures (e.g., LSTM $\to$ TCN) can substantially compensate for individual weaknesses. For instance, an LSTM followed by a TCN reduces sensitivity to Impulse targets by five orders of magnitude compared to standalone models. The ordering matters, with specific architectures better suited for feature extraction versus refinement.
Mixed Targets: When targets combine multiple memory structures (e.g., Exponential + Impulse), specialized architectures degrade sharply if they cannot handle one component. Transformers show greater robustness in these hybrid scenarios, exhibiting non-monotonic loss patterns that allow recovery as one component dominates.

Significance and Claims

The paper claims that its proposed framework provides a principled methodology for isolating and evaluating specific temporal properties of sequence models. By using controllable synthetic targets, the authors:

Validate existing theories: Confirming that RNNs struggle with polynomial decay and that TCNs excel at sparse dependencies.
Reveal new phenomena: Identifying the critical role of depth in sparse dependency modeling, the tail-energy complexity in TCN approximation, and the specific head-dimensionality trade-offs in Transformers.
Highlight architectural trade-offs: Demonstrating that no single architecture excels across all temporal structures, suggesting that architecture selection must be guided by the specific temporal dependencies of the target data.
Propose hybrid strategies: Showing that sequential composition can effectively bridge the gaps between architectural limitations.

The authors conclude that while their analysis focuses primarily on approximation properties with scalar targets, the framework is extensible to optimization, generalization, and multivariate sequences, offering a flexible tool for advancing the theoretical understanding of sequence modeling.