TimeMM: Time-as-Operator Spectral Filtering for Dynamic Multimodal Recommendation

1. Problem Statement

Multimodal recommendation systems aim to improve user modeling by integrating collaborative signals with heterogeneous item content (e.g., images and text). However, existing approaches face two critical limitations in real-world applications:

• Static Temporal Assumptions: Most state-of-the-art methods treat user-item interactions as static bipartite graphs, aggregating historical data indiscriminately. This fails to capture the non-stationary dynamics of user interests, where different preference factors (e.g., visual style vs. semantic category) evolve at different rates.
• Rigid Spectral Modeling: While recent spectral graph methods decompose signals into frequency bands to separate noise from trends, they typically rely on explicit eigendecomposition of static graphs. This results in fixed spectral bases that cannot adapt to time-varying shifts in user behavior. Furthermore, they often apply uniform filtering across modalities, ignoring that visual cues may react faster to trends than textual semantics.

The core challenge is to model continuous preference evolution with fine-grained temporal adaptation and modality-specific sensitivity without incurring the high computational cost of explicit spectral decomposition.

2. Methodology: TimeMM

The authors propose TimeMM, a framework that treats time as an operator to induce spectral filtering implicitly. The architecture consists of four key components:

A. Time-as-Operator (Temporal Kernels as a Spectral Filter Bank)

Instead of appending time as a feature, TimeMM maps interaction recency directly into the graph propagation operator.

• Temporal Proximity Kernel: Interaction recency ($\Delta t$) is mapped to edge weights using a parametric kernel: $w(\Delta t) = \exp(-\log(1+\Delta t)/\tau)$.
• Spectrum of Propagation Scales: A set of temporal scales $\{\tau_1, ..., \tau_K\}$ is defined. Each scale generates a distinct weighted adjacency matrix, creating a lightweight spectral filter bank.
  • Small $\tau$ (Short-scale): Emphasizes recent interactions (high-frequency signals, transient trends).
  • Large $\tau$ (Long-scale): Preserves long-range history (low-frequency signals, stable habits).
• Parallel Propagation: Message passing (based on LightGCN) is performed in parallel across these $K$ scales. This avoids explicit eigendecomposition, maintaining linear time complexity $O(KL|E|D)$.

B. Adaptive Spectral Filtering

Since the optimal spectral profile varies per user and item, TimeMM learns to mix the outputs of the $K$ operators dynamically.

• Temporal Context State: Compact vectors ($s_u, s_i$) are constructed from interaction history statistics (e.g., activity span, mean interaction age).
• Gating Mechanism: A learnable gating network maps these context states to mixture weights ($g_u, g_i$) over the $K$ operators. This allows the model to select an "effective spectral response" specific to the current prediction context (e.g., prioritizing short-term signals for a user with volatile interests).

C. Spectral-Aware Modality Routing

Recognizing that different modalities have different temporal sensitivities, TimeMM routes modality contributions based on the active spectral regime.

• Context-Conditioned Routing: A lightweight encoder maps the temporal context to modality weights ($\beta$).
• Dynamic Fusion: Visual and textual features are re-weighted and concatenated. For instance, short-range regimes may emphasize visual cues (fast-changing), while long-range regimes prioritize textual semantics (stable).

D. Spectral Diversity Regularization

To prevent the learned operator bank from collapsing into redundant experts (filter-bank collapse), a regularization term is applied in the ranking space.

• It enforces diversity in the preference margins of different spectral components, ensuring that each operator learns a complementary smoothing profile rather than averaging out to the same behavior.

3. Key Contributions

1. Time-as-Operator Paradigm: A novel approach that converts interaction timestamps into a family of parametric temporal kernels, creating a spectral filter bank implicitly without expensive eigendecomposition.
2. Adaptive Framework: The integration of Adaptive Spectral Filtering (context-dependent operator mixing) and Spectral-Aware Modality Routing (context-dependent modality weighting) to handle non-stationary, multi-scale user dynamics.
3. Scalability: The method achieves linear-time scalability with respect to graph size, making it suitable for large-scale industrial applications, unlike explicit spectral methods which are cubic in complexity.
4. Theoretical Interpretation: The paper provides a theoretical link showing that the proposed time-weighted propagation behaves as polynomial spectral filtering, where different temporal scales correspond to different frequency responses.

4. Experimental Results

The model was evaluated on four Amazon benchmark datasets (CD, Game, Baby, Software) and a real-world industrial offline dataset.

• Overall Performance: TimeMM consistently outperformed state-of-the-art baselines (including static multimodal models like AlignRec, MMGCN, and spectral models like SMORE, FITMM).
  • It achieved relative improvements of 3%–10% in Recall@K and NDCG@K over the strongest baselines.
  • On the Industrial dataset, it improved Recall@10 from 0.1221 (best baseline) to 0.1271.
• Ablation Studies: Removing the "Time-as-Operator" component caused the largest performance drop, confirming that injecting temporal dynamics into the propagation operator is the primary driver of success.
• Timestamp Sensitivity: Perturbation experiments showed that TimeMM is highly sensitive to the relative order of timestamps. Shuffling timestamps caused significant performance degradation, proving the model relies on genuine temporal structure rather than incidental correlations.
• User Span Analysis: The performance gain of TimeMM over static spectral baselines increased with the user's history span. Long-span users (richer temporal structure) benefited the most, validating the model's ability to capture long-horizon drift.
• Operator Behavior: Analysis confirmed that the learned operators follow a consistent short-to-long ordering (monotonic energy decay), and the modality routing shifts systematically (e.g., visual cues dominate for short-span users, text for long-span users).

5. Significance

This paper represents a significant advancement in dynamic recommendation systems by bridging the gap between spectral graph theory and temporal dynamics.

• Theoretical Shift: It moves away from the "static graph + time feature" paradigm to a "time-as-operator" paradigm, treating time as an intrinsic property of the graph topology.
• Practical Impact: By avoiding explicit eigendecomposition, it solves the scalability bottleneck of spectral methods, making advanced frequency-domain modeling feasible for industrial-scale systems with millions of users and items.
• Modality Insight: It demonstrates that multimodal signals are not static; their relevance is dynamically conditioned on the temporal regime of the user, offering a more nuanced understanding of user intent.

The code is open-sourced, facilitating further research in dynamic graph learning and multimodal recommendation.