🤖 machine learning

TIDES: Implicit Time-Awareness in Selective State Space Models

The paper proposes TIDES, a selective state space model variant that preserves the physical meaning of time discretization for native irregular time series handling while maintaining high per-token expressivity by shifting input-dependence from the step size to the state matrix, achieving state-of-the-art performance on time-series benchmarks.

Original authors: Taylan Soydan, Miguel A. Bessa, Dirk Mohr, Rui Barreira

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Taylan Soydan, Miguel A. Bessa, Dirk Mohr, Rui Barreira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a story, but the story is told in a very strange way. Sometimes the narrator speaks every second, sometimes they wait five minutes, and sometimes they skip days entirely. This is what happens with irregular time-series data (like a patient's heart rate monitor that only beeps when something changes, or a weather sensor that stops working for hours).

To handle this, the robot needs two superpowers:

Context Awareness: It needs to know what is happening right now to decide if it should remember the past or forget it. (e.g., "Oh, a heart attack just happened? I better remember this!" vs. "Just a normal heartbeat? I can forget the last one.")
Time Awareness: It needs to understand how much time passed between events. (e.g., "It's been 5 seconds since the last beep" vs. "It's been 5 hours.")

The Problem with Previous Robots

The paper argues that the two best robots built so far (called Mamba and S5) each have one of these superpowers but are missing the other.

The "Mamba" Robot: This robot is great at Context Awareness. It can look at a specific word or event and decide, "I need to pay extra attention to this!" However, to do this, it treats time as a flexible, made-up number. It learns a "fake time" based on the input.
- The Flaw: If you train this robot on stories where gaps are usually 1 minute, and then you ask it to read a story with gaps of 1 hour, it gets confused. It tries to apply its "fake time" logic to a situation it's never seen before and fails. It's like a chef who learned to cook only when the timer says "5 minutes," but when the timer says "5 hours," they don't know how to adjust the heat.
The "S5" Robot: This robot is great at Time Awareness. It treats time exactly like a real clock. If 5 hours pass, it knows 5 hours have passed, no matter what.
- The Flaw: It is too rigid. It treats every input the same way. It can't say, "This specific event is important, so I'll remember it longer." It's like a robot that counts seconds perfectly but has no brain to decide what is worth remembering.

The Solution: TIDES

The authors created a new robot called TIDES (Time-Implicit Decay and Eigenvalue Selectivity).

Think of TIDES as a hybrid chef.

The Clock (Time): TIDES keeps a real, physical clock. It never fakes the time. If the gap between events is 5 hours, the robot knows it is 5 hours. It doesn't try to learn a "fake time" from the input. This allows it to handle weird, irregular gaps perfectly, even if it has never seen that specific gap size before.
The Brain (Selectivity): Instead of faking the time to be smart, TIDES changes its internal memory settings based on the input. When a big event happens, it tells its internal memory, "Slow down the forgetting process for this specific piece of information." When a boring event happens, it says, "Forget this quickly."

The Magic Trick:
In previous models, the robot tried to be smart by changing the clock speed (the time step). TIDES keeps the clock speed real and physical, but changes the memory decay rate (how fast it forgets). This way, it gets the best of both worlds: it understands the real passage of time and it knows what to remember.

The "Fading Flash" Test

To prove this works, the authors invented a simple game called "Fading Flash."
Imagine a row of 40 light detectors.

A flash happens.
The detector glows and then slowly fades away.
The speed at which it fades depends on which "zone" of the row it is in (Slow, Medium, or Fast).
The time between flashes is random and irregular.

The Old Robots: The "S5" robot couldn't learn the different fading speeds for different zones. The "Mamba" robot could learn the speeds, but if you changed the time between flashes to something it hadn't seen in training, it completely failed.
The TIDES Robot: It learned the different fading speeds for every zone and it worked perfectly even when the time between flashes was totally new and weird.

The Results

The authors tested TIDES on real-world datasets (like medical sensor data and biological simulations).

It became the new champion (State-of-the-Art) for classifying time-series data.
It became the new champion for predicting biological equations.
Crucially, it didn't just get lucky on the training data; it handled "out-of-distribution" data (weird, unseen time gaps) much better than its competitors.

Summary

TIDES is a new type of AI that stops trying to "fake" time to be smart. Instead, it keeps time real and physical, but makes its memory flexible. This allows it to handle messy, irregular real-world data (like sensors that drop out or medical records that arrive late) without getting confused when the timing changes.

Technical Summary: TIDES – Implicit Time-Awareness in Selective State Space Models

Problem Statement

Real-world sequential data, such as clinical observations, environmental sensor readings, and financial events, is rarely sampled on a regular grid. Standard sequence models often assume uniform time steps, forcing them to either resample data (discarding temporal information) or augment inputs with explicit time features.

Two dominant families of State Space Models (SSMs) address sequence modeling but face a fundamental trade-off regarding irregular time series:

Continuous-time SSMs (e.g., S5): These preserve the physical meaning of the time step ( $\tilde{\Delta} \equiv \Delta$ ), allowing them to handle irregular timestamps natively by integrating the continuous dynamics over the actual elapsed time. However, they are typically Linear Time-Invariant (LTI), meaning their dynamics do not change based on the input content, limiting per-token expressivity.
Selective SSMs (e.g., Mamba): These achieve strong per-token expressivity by making the discretization step $\tilde{\Delta}$ and projection matrices ( $B, C$ ) input-dependent functions. This allows the model to adapt its effective time constant token-by-token. However, this design collapses $\tilde{\Delta}$ from a physical sampling interval into a learned gate. Consequently, the model loses its native handling of irregular time; to process irregular data, it must treat time deltas as explicit input features. The paper argues this indirect coupling prevents the model from extrapolating to irregularity regimes (time deltas) unseen during training.

Methodology: TIDES

The authors propose TIDES (Time-Implicit Decay and Eigenvalue Selectivity), a selective SSM variant designed to reconcile the expressivity of Mamba with the physical time semantics of S5.

Core Design Principle

TIDES moves input-dependence from the discretization step ( $\tilde{\Delta}$ ) to the continuous-time generator (the state matrix $\Lambda$ and projections $B, C$ ).

Implicit Time-Awareness: The time step $\tilde{\Delta}$ remains strictly physical ( $\tilde{\Delta} \equiv \Delta$ ), determined solely by the observation timestamps. It is not a learned function of the input.
Selective Generator: Input-dependence is applied to:
- $\text{Re}(\Lambda)$ : The real part of the eigenvalues (decay rates) becomes a function of the input. This allows the model to "forget faster" or "hold longer" based on content, providing the necessary selectivity.
- $B$ and $C$ : The input and output projections are input-dependent, allowing the model to filter noise and modulate state reading/writing.
- $\text{Im}(\Lambda)$ : The imaginary part (oscillation frequency) remains static. The authors argue that modulating frequency per-token lacks a clean continuous-time interpretation and empirically find it hurts performance.

Architecture Details

Initialization: The model initializes with HiPPO-derived biases for $\Lambda, B, C$ and zero-initialized projection weights. This ensures the model starts from a well-conditioned LTI recurrence and learns selectivity as a smooth perturbation.
Low-Rank Projections: To manage parameter counts, the dense projections for $B$ and $C$ are factorized into low-rank structures ( $r \ll PH$ ).
Deep Projectors: The input pathway includes residual Gated Linear Unit (GLU) blocks before the final projection to introduce non-linearity, similar to Mamba's preprocessing.
Normalization: RMSNorm is applied to the projected parameters ( $\text{Re}(\Lambda), B, C$ ) immediately before discretization to stabilize training, preventing unbounded perturbations in the recurrence.
Stable Reparameterization: $\text{Re}(\Lambda)$ is reparameterized (e.g., via exponential or stable mapping) to ensure eigenvalues remain in the stable half-plane, preventing explosive recurrences regardless of the input.

Key Contributions

Architecture: TIDES introduces a design where selectivity resides in the continuous-time generator ( $\text{Re}(\Lambda), B, C$ ) while the discretization step remains a physical, implicit quantity. This allows the model to handle irregular timestamps natively without sacrificing per-token expressivity.
Mechanistic Analysis (Fading Flash): The authors introduce a novel diagnostic benchmark, Fading Flash, which isolates two failure modes:
- LTI Rigidity: The inability of S5 to model multiple decay rates simultaneously.
- Extrapolation Failure: The inability of Mamba-like models to generalize to time deltas ( $\Delta$ ) outside the training distribution.
- TIDES is shown to be the only architecture that overcomes both, maintaining robust performance across unseen $\Delta$ values while capturing zone-dependent dynamics.
Empirical Results: TIDES achieves new state-of-the-art (SOTA) average ranks on:
- UEA Time-Series Classification: Outperforming LogNCDE, RFormer, and other baselines.
- Physiome-ODE Regression: Surpassing LinODEnet and other irregular time-series forecasting models.

Experimental Findings

Fading Flash: In this controlled experiment, S5 failed to capture zone-dependent decays (low expressivity), while MambaS (a Mamba surrogate) failed to extrapolate to test-time $\Delta$ values outside the training range (drifting effective decay rates). TIDES successfully learned the correct decay rates for all zones and remained robust across a wide range of test-time $\Delta$ .
Random Drop Ablation: On the EigenWorms dataset with random time-step dropping, TIDES maintained high accuracy even at extreme drop rates ( $r_{test}=0.9$ ) unseen during training. In contrast, MambaS and RFormer collapsed significantly at high drop rates, confirming that input-dependent $\tilde{\Delta}$ or fixed signature windows fail to generalize to irregular sampling regimes.
Ablation on $\text{Im}(\Lambda)$ : Making the imaginary part of eigenvalues input-dependent did not improve performance and slightly degraded it, supporting the design choice to keep oscillation frequencies static.

Significance and Claims

The paper claims that TIDES demonstrates that per-token expressivity and faithful physical-time semantics are compatible, provided that input-dependence is routed through the continuous-time generator rather than the discretization step.

The authors posit a broader design principle for irregularly-sampled sequences: keep discretization and expressivity separate. The physical sampling interval should enter the model only through the analytic discretization rule, while expressivity should reside in the continuous-time generator. This separation enables models to be simultaneously expressive and extrapolative to sampling regimes not seen during training.

The paper is modest in its scope, explicitly stating that TIDES is designed for irregular time series (clinical, sensor, simulation data) and is not claimed as a general replacement for Mamba on regular-grid tasks like language modeling, where the role of the input-dependent time gate differs. The work suggests that the principle of separating time-handling from expressivity may extend to other continuous-time architectures, including Neural ODEs and continuous-time attention variants.