🤖 machine learning

Continuity Laws for Sequential Models

This paper formalizes and empirically validates the concept of temporal continuity in sequential models, demonstrating that models like S4 exhibit stable continuous behavior aligned with task continuity for improved performance, whereas models like Mamba (S6) are more sensitive to discretization, ultimately showing that leveraging continuity enables more efficient and effective temporal subsampling strategies.

Original authors: Annan Yu, Dongwei Lyu, N. Benjamin Erichson

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Annan Yu, Dongwei Lyu, N. Benjamin Erichson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. Some things in the world happen in smooth, flowing waves (like a river or a heartbeat), while others happen in distinct, choppy steps (like typing a sentence or flipping a light switch).

This paper asks a simple but profound question: Do the AI models we build actually "feel" the difference between smooth and choppy, or are they just pretending?

Here is the breakdown of their findings using everyday analogies:

1. The "Smoothness" Test

The authors looked at two popular types of AI models: S4 and S6 (the brain behind the famous "Mamba" model). Both are built using math that says they are designed to handle smooth, continuous time.

The Analogy: Imagine you are drawing a curve.
- S4 is like a skilled artist with a steady hand. If you ask them to draw the curve using fewer dots (coarse sampling) and then more dots (fine sampling), the picture stays smooth and consistent. The dots just get closer together, but the line doesn't jump or wiggle.
- S6 is like a robot arm that is supposed to draw a smooth line, but it gets jittery if you press the pen too hard or if the line changes direction quickly. When you zoom in (refine the time), S6's drawing starts to look shaky and inconsistent. It claims to be smooth, but in practice, it behaves more like a choppy, step-by-step machine.

The Finding: Just because a model is designed with continuous math doesn't mean it acts continuously. S4 actually behaves smoothly; S6 is sensitive to how loud or strong the input is, making it less "continuous" in reality.

2. Matching the Tool to the Job

The paper introduces a new rule: You need a smooth tool for a smooth job, and a choppy tool for a choppy job.

The Analogy: Think of a smoothie (continuous data like weather or stock prices) vs. a staircase (discrete data like text or code).
- If you try to blend a staircase with a blender (a smooth model), you get a mess.
- If you try to climb a smoothie with stairs (a choppy model), you can't get a grip.
The Experiment: The researchers created tasks that were half-smooth and half-choppy. They found that:
- When the task was mostly choppy (like text), the "choppy" models (S6, Transformers) did better.
- When the task was mostly smooth (like physics simulations), the "smooth" model (S4) crushed the competition.

The Finding: Performance isn't just about how big the model is; it's about whether the model's "personality" (its bias toward smoothness or steps) matches the "personality" of the data.

3. The "Zoom-In" Training Trick

Because S4 is truly smooth, the authors discovered a clever way to train it faster.

The Analogy: Imagine learning to drive a car.
- Standard Training: You start driving immediately on a busy, high-speed highway with full details. It's stressful and slow to learn the basics.
- The Paper's Trick: You start driving on a wide, empty dirt road at low speed (low resolution). You learn the basics of steering and stopping. Then, you gradually move to a paved road, then a city street, and finally the highway, getting faster and more detailed at each step.
How it works: They trained the S4 model on data that was "subsampled" (skipping every few seconds of data). Because S4 is smooth, it could learn the general shape of the story from the skipped data. Then, they slowly filled in the missing seconds.
The Result: This method made training much faster (less computer time) and, surprisingly, sometimes even more accurate than training on the full data from the start.

Crucial Note: This trick only worked for S4. When they tried it on S6 (the "jittery" model), the model got confused at every step because it couldn't handle the jump from "low resolution" to "high resolution." This proves that S6 isn't truly continuous.

Summary

The Problem: We assume some AI models are "continuous" because of their math, but we didn't check if they actually act that way.
The Discovery: S4 is genuinely smooth; S6 is sensitive and acts more like a step-function.
The Rule: Smooth models work best on smooth data (physics, time series); choppy models work best on discrete data (text, symbols).
The Bonus: If you have a truly smooth model, you can train it by starting with a "blurry" version of the data and sharpening it over time, saving huge amounts of time and money.

The paper concludes that understanding this "continuity" helps us pick the right AI for the right job and gives us a new, faster way to train them.

Technical Summary: Continuity Laws for Sequential Models

Problem Statement

While sequential models (e.g., Transformers, State-Space Models) have achieved strong performance on specific benchmarks, their behavior often lacks a principled understanding regarding why they succeed or fail in different temporal regimes. Existing principles, such as neural scaling laws, describe performance changes with model size but do not address whether a model's structural inductive bias matches the temporal structure of the task. Specifically, there is a gap in understanding whether models motivated by continuous-time formulations (like State-Space Models or SSMs) actually exhibit continuous behavior when discretized, and whether this "continuity" is a useful inductive bias for tasks with continuous temporal structure (e.g., dynamical systems) versus discrete tasks (e.g., text).

Methodology

1. Formalizing Model Continuity

The authors define model continuity through the lens of convergence under temporal refinement. A model is considered continuous if, as the temporal discretization step size $\tau$ approaches zero, its discrete predictions converge to a common underlying continuous trajectory at a stable rate.

Theoretical Framework: The paper derives convergence bounds for general continuous-time dynamical systems discretized via Zero-Order Hold (ZOH) or bilinear methods.
Comparison of SSMs: The framework is applied to compare S4 (linear time-invariant) and S6/Mamba (input-dependent, selective).
- S4: Theoretical analysis (Corollary 1) shows the convergence error depends only on fixed system parameters and input regularity.
- S6: Theoretical analysis (Corollary 2) reveals the error constant is highly sensitive to input amplitude ( $M_u$ ), projection magnitudes, and selectivity parameters. This suggests S6's discrete behavior is less stable under refinement.
Empirical Validation: Numerical experiments on 20 random system-input pairs confirm that S4 error remains stable under input scaling and temporal refinement, whereas S6 error increases significantly with input magnitude.

2. Quantifying Task Continuity

To measure the continuity of a dataset, the authors introduce a data-driven metric ( $\mu$ ) based on local temporal similarity.

Definition: The metric compares the similarity of nearby elements in a sequence against far-apart elements. It aggregates local similarity scores over a range of lags, subtracting a background similarity term ( $\beta$ ) to isolate local continuity.
Application: This metric places datasets on a spectrum from discrete (e.g., text, symbolic sequences) to continuous (e.g., images, dynamical system trajectories).

3. Investigating Model-Task Alignment

The authors conduct controlled experiments to test the hypothesis that model continuity should align with task continuity.

Synthetic Continuity Control: They construct a forecasting task where the input representation is a weighted interpolation between a continuous embedding and a quantized (discrete) embedding. By varying the weight $\eta$ , they create a continuum of task difficulty regarding continuity.
Benchmark Evaluation: They evaluate S4, S6, and Transformers on the Long-Range Arena (LRA) benchmarks, computing the $\mu$ score for each task.

4. Algorithmic Application: Stage-wise Training

Leveraging the observation that continuous models (like S4) are stable under temporal refinement, the authors propose a stage-wise training strategy.

Mechanism: Training begins with temporally subsampled (coarse) inputs. The model is trained on this low-resolution data, and the resolution is progressively refined in subsequent stages. The discretization parameter $\Delta$ is scaled accordingly at each stage.
Rationale: This exploits the model's ability to learn coarse patterns first, acting as a form of implicit regularization and reducing early training costs.

Key Results

Discrepancy Between Motivation and Behavior: While S4, S5, and S6 all originate from continuous-time formulations, only S4 and S5 exhibit stable continuous behavior under temporal refinement. S6 (Mamba) is significantly more sensitive to input amplitude and selective dynamics, causing its discrete predictions to diverge from the continuous trajectory as resolution increases.
Alignment of Continuity and Performance:
- Synthetic Tasks: As tasks become more continuous (higher $\eta$ ), S4 outperforms S6 and Transformers. Conversely, on discrete tasks (low $\eta$ ), S6 and Transformers dominate.
- LRA Benchmarks: Tasks with higher computed continuity scores (e.g., Image, Pathfinder, PathX) show a clear performance advantage for S4 over S6 and Transformers. Tasks with lower continuity scores (e.g., ListOps, Text) favor S6 and Transformers.
Efficiency Gains: The stage-wise training strategy significantly accelerates training for continuous models. On five of six LRA tasks, the subsampled S4D model reaches target accuracies in less wall-clock time than standard training. On highly continuous tasks, it also achieves higher final accuracy.
Selectivity of the Strategy: The subsampling strategy fails for models lacking continuous behavior (e.g., S6), where stage transitions cause sharp drops in accuracy, confirming that the method relies on the underlying model's continuity properties.

Significance and Claims

The paper posits continuity as a fundamental inductive bias that governs the fit between sequential models and their tasks.

Diagnostic Value: Continuity serves as a principle to predict model performance: models with strong temporal continuity are better suited for continuous tasks (dynamical systems, time series), while models with weaker continuity may be better for discrete tasks (language, symbolic operations).
Distinction from Scaling: Unlike scaling laws which address capacity, continuity laws address structural compatibility. A mismatched architecture cannot be fixed solely by scaling; the inductive bias must align with the task's temporal structure.
Practical Utility: Continuity is not merely a diagnostic property but a structural feature that can be exploited algorithmically. The proposed stage-wise training demonstrates that understanding a model's continuity can lead to more efficient training procedures without modifying the model architecture itself.

The authors conclude that continuity is a "law-like organizing principle" for sequential modeling, suggesting that future work should explore how continuity interacts with other inductive biases (e.g., frequency, recency) to further refine model selection and design.