Technical Summary: Linear Recurrent Unit with Semantic Modulation for Image Super-Resolution

1. Problem Statement

Image Super-Resolution (SR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While Transformer-based models have achieved high-quality reconstructions by capturing long-range dependencies, they often suffer from high computational complexity. Recent State-Space Models (SSMs), such as Mamba, offer computational efficiency and global receptive fields but rely on dynamic parameterization and intricate discretization procedures that increase model complexity and reduce interpretability.

Conversely, Linear Recurrent Units (LRUs) offer a simplified, stable, and efficient alternative to deep SSMs with linear complexity. However, standard LRUs employ static parameterization and a single-scan method. This static nature limits their adaptability to the complex, non-linear, and spatially varying patterns inherent in 2D vision tasks like SR, where both long-range context and fine local details are critical. Existing methods attempting to improve multi-scan properties often resort to dynamic scanning, which compromises resource efficiency.

2. Methodology: LSM (Linear Recurrent Unit with Semantic Modulation)

The authors propose LSM, a novel SR network that unifies the stability of LRUs with a Semantic Modulating Unit (SMU). The core philosophy is to maintain the computational efficiency of a static scan while introducing dynamic, input-dependent adaptivity through semantic modulation.

2.1. Core Architecture

The network follows a hierarchical structure similar to Griffin (a hybrid of gated linear recurrences and local attention), consisting of:

• Local Blocks: Window-based Multi-head Self-Attention (WMS) blocks.
• Global Blocks: Category-based Modulated LRU (CML) blocks.
• Residual Connections: Adaptive skip connections with learnable weights.

2.2. The Semantic Modulating Unit (SMU)

The SMU is the central innovation, designed to overcome the limitations of vanilla LRU. It performs three distinct, complementary roles:

1. LRU Modulation (Input-Dependent Gating):

   • Unlike vanilla LRU, where transition matrices ($\lambda, B, C$) are static, the SMU generates modulating tokens ($M_k$) based on input features and a learned dictionary.
   • These tokens dynamically modulate the LRU's hidden state update equation. Specifically, they adjust the eigenvalue scaling ($\lambda$), input projection ($B$), and output projection ($C$) on a pixel-wise basis.
   • This allows the recurrence dynamics to adapt to the specific content of the input, effectively injecting non-linearity and context-awareness without the overhead of dynamic scanning.

2. Spatial Categorization:

   • Standard 1D recurrence struggles to relate spatially distant but semantically similar pixels.
   • The SMU assigns pixels to one of $P$ semantic groups using a temperature-controlled Gumbel softmax over a learned dictionary of prototype tokens.
   • This categorization reorganizes the input sequence, allowing semantically related but distant pixels to interact within a single scan, thereby enhancing the coherence of the input sequence for the LRU.

3. Feature Enhancement (Cross-Attention):

   • In parallel with modulation, the SMU performs cross-attention over the learned dictionary.
   • It aggregates globally relevant information across the entire spatial domain, producing an enhanced feature map ($Y_{enhance}$).
   • This map is concatenated with the LRU output ($Y_{LRU}$), providing complementary global context that a single-scan recurrence might miss.

2.3. Initialization and Training

• Eigenvalue Initialization: The paper investigates the initialization of LRU eigenvalues ($\lambda$) in the complex plane. The optimal setting involves a ring-shaped distribution with specific magnitude ($r_{min}, r_{max}$) and phase ($\theta_{max}$) ranges to balance local pattern preservation and global consistency.
• Dictionary Learning: The SMU utilizes a compact learned dictionary ($D \in \mathbb{R}^{P \times c}$) where $P \ll T$ (number of tokens), ensuring parameter efficiency.

3. Key Contributions

1. First LRU-based SR Network: The authors introduce LSM as the first network to apply Linear Recurrent Units to image super-resolution, successfully unifying linear recurrence with semantic modulation.
2. Semantic Modulating Unit (SMU): A novel module that serves three roles:
   • Modulates LRU transitions via input-dependent gating.
   • Categorizes pixels by semantic similarity to construct coherent input sequences.
   • Enhances feature representation via cross-attention over a learned dictionary.
3. Efficiency-Performance Trade-off: The method demonstrates that the LRU backbone's resource efficiency allows for allocating more capacity to the SMU, achieving superior performance under limited computational resources compared to dynamic SSMs and heavy Transformers.

4. Experimental Results

The authors evaluated LSM on standard benchmarks (Set5, Set14, B100, Urban100, Manga109) across $\times2, \times3, \times4$ scaling factors.

• Quantitative Performance:
  • LSM variants (LSM-S, LSM, LSM+) consistently outperform state-of-the-art CNN, Transformer, and Mamba-based methods.
  • On the $\times4$ Urban100 dataset, LSM-S surpasses MambaIRv2-S by 0.15 dB PSNR.
  • In lightweight settings, LSM-light outperforms MambaIRv2-light and OmniSR, showing significant gains on high-frequency datasets like Manga109 and Urban100.
• Computational Efficiency:
  • Compared to MambaIR, LSM reduces FLOPs by 32.8% while improving PSNR by 0.26 dB on Urban100.
  • Compared to Transformer-based CAT-A, LSM reduces parameters by 41.2% and FLOPs by 44.8% while achieving higher PSNR.
  • The model exhibits superior memory efficiency during training, allowing for larger input resolutions under fixed hardware budgets.
• Ablation Studies:
  • Initialization: Proper eigenvalue initialization is critical; suboptimal settings lead to performance drops of up to 0.19 dB.
  • SMU Roles: Removing any of the three SMU roles (categorization, cross-attention, modulation) results in performance degradation, confirming the synergy of the multi-role design.
  • Modulation Tokens: The coupled design of modulation tokens and cross-attention (sharing a softmax branch) proves more effective than decoupling them, balancing parameter efficiency and performance.

5. Significance and Claims

The paper claims that LSM represents a significant step forward in balancing efficiency and performance for image restoration. By leveraging the stability of LRUs and enhancing them with a lightweight, semantic-aware modulation mechanism, the authors demonstrate that:

• Static scanning can be made adaptive to complex 2D visual patterns without resorting to computationally expensive dynamic scanning.
• The integration of semantic categorization and dictionary-driven priors allows the model to capture both long-range contextual information and detailed local textures effectively.
• LSM offers a competitive, next-generation backbone for SR that achieves state-of-the-art results with significantly lower computational complexity and memory usage compared to existing Transformer and Mamba-based approaches.

The authors conclude that their work provides a principled approach to adapting linear recurrence for vision tasks, suggesting strong potential for future applications in low-level image processing where resource constraints are critical.