Technical Summary: MRT – A Mamba-based Framework with Reusable Transformer for Scene Text Recognition

1. Problem Statement

Scene Text Recognition (STR) requires models to simultaneously resolve two conflicting challenges: preserving fine-grained local details (character stroke structures and boundaries) and capturing long-range global semantic dependencies to disambiguate distorted or occluded text.

While Convolutional Neural Networks (CNNs) excel at local feature extraction, their limited receptive fields struggle with long-range dependencies. Conversely, Vision Transformers (ViTs) capture global context via self-attention but suffer from quadratic computational complexity, leading to high memory consumption and inference latency. Recently, State Space Models (SSMs), specifically Mamba, have emerged as efficient alternatives with linear-time complexity. However, the authors identify a critical structural mismatch when applying vanilla Mamba to STR:

1. Causal Scanning: Mamba's sequential, causal scanning is unidirectional, failing to capture the bidirectional nature of language and the 2D spatial locality of visual text features.
2. Lack of Spatial Locality: The original formulation is sequence-centric and does not explicitly encode the 2D spatial locality essential for recognizing fine-grained character glyphs.

These limitations result in ineffective local feature extraction and suboptimal global context modeling, decoupling efficiency gains from recognition accuracy.

2. Methodology

The authors propose MRT (Mamba-based framework with Reusable Transformer), a hierarchical architecture designed to reconcile Mamba's efficiency with the specific demands of STR. The framework consists of three main components:

A. NA-Mamba (Neighborhood Attention Mamba)

To address the structural mismatch of vanilla Mamba, the authors introduce NA-Mamba, which modifies the SSM pathway in two ways:

• Removal of Causality: The original causal 1D convolution is replaced with a standard (non-causal) 1D convolution. This allows the current token to access both preceding and succeeding local contexts, enabling bidirectional local interactions crucial for resolving character ambiguities.
• Parallel Neighborhood Attention (NA) Branch: A parallel branch is added alongside the SSM pathway. This branch performs content-adaptive aggregation within a local neighborhood window ($\Omega(i)$) rather than attending globally. This explicitly strengthens fine-grained text structure modeling at a lower computational cost than global self-attention, which is particularly beneficial for irregular layouts and perspective distortions.
• Fusion: The outputs of the SSM and NA branches are projected to match dimensions, fused via element-wise addition, and passed through a residual connection.

B. Reusable Transformer

To compensate for the lack of bidirectional global context in Mamba, the authors incorporate a Reusable Transformer module.

• Placement: This module is applied after the second and third stages of the feature extraction hierarchy.
• Parameter Sharing: A single Transformer encoder (parameterized by $\theta$) is reused across these stages. Instead of using independent Transformer blocks for each stage, the same weights are applied to features from different stages.
• Architecture: The Transformer follows a standard encoder paradigm but replaces the standard residual connection after self-attention with a Squeeze-and-Excitation (SE) block for channel-wise feature recalibration.
• Benefits: This design drastically reduces parameter overhead while enforcing consistency in global feature interactions and accelerating training convergence.

C. Overall Architecture

MRT adopts a three-stage hierarchical design:

1. Patch Embedding: Converts input images into spatially aligned visual tokens.
2. Feature Extraction: Three stages, each containing a downsampling module (MBConv) and multiple NA-Mamba layers.
3. Refinement: Reusable Transformer blocks are inserted after stages 2 and 3.
4. Decoding: Refined features are compressed and fed into a pre-normalized Transformer decoder with dual-stream attention to generate the prediction sequence.

3. Key Contributions

The paper outlines three primary contributions:

1. Tailored Adaptation of Mamba for STR: The proposal of NA-Mamba, which integrates standard convolutions and a parallel Neighborhood Attention branch to address Mamba's deficiencies in local feature extraction and unidirectional scanning.
2. Design of Reusable Transformer: The introduction of a parameter-sharing strategy where a single Transformer encoder is reused at multiple stages. This strengthens global dependency modeling while significantly reducing parameter counts and accelerating convergence.
3. The MRT Framework: A complete backbone that diverges from progressive downsampling paradigms to preserve feature scales aligned with the reusable module. It serves as a viable alternative to ViTs and CNNs, achieving competitive performance with substantially fewer parameters.

4. Experimental Results

The authors evaluated MRT on six standard benchmarks (IIIT5K, SVT, IC13, IC15, SVTP, CUTE80) and the large-scale Union-14M benchmark.

• Performance vs. Efficiency:
  • MRT-B (Base): Achieves 96.96% average accuracy on standard benchmarks, outperforming comparably sized or larger models (e.g., SVTRv2, SMRT) while maintaining a similar or smaller parameter footprint (23.48M).
  • MRT-T (Tiny): A lightweight variant with only 4.72M parameters achieves 95.06% average accuracy, surpassing or matching several recent methods with significantly larger capacities.
• Union-14M Benchmark: MRT-B achieves 84.76% average accuracy, outperforming or matching larger models like ABINet and LISTER, particularly on challenging subsets such as Artistic, Salient, and Multi-Oriented text.
• Ablation Studies:
  • NA-Mamba: Replacing causal convolutions and adding the NA branch improved average accuracy by 0.5%, with significant gains (up to 1.7%) on irregular text datasets (e.g., SVTP).
  • Reusable Transformer: Introducing the Transformer improved accuracy by ~2% over the baseline. Parameter sharing (Reuse) achieved nearly the same accuracy as non-shared Transformers (0.22% difference) but with a 17.8% reduction in model size.
  • Convergence: The Reuse configuration demonstrated faster loss reduction and quicker convergence during the early training phase compared to non-shared configurations.

5. Significance and Claims

The paper claims that MRT demonstrates that Mamba-based architectures can serve as competitive and efficient backbones for STR when equipped with task-appropriate spatial modeling and lightweight global refinement.

• Balancing Act: The framework successfully addresses the inherent tension between localized precision and holistic contextual reasoning, achieving a strong accuracy-efficiency trade-off.
• Efficiency: By avoiding over-parameterization through the Reusable Transformer and optimizing local modeling via NA-Mamba, MRT facilitates more efficient optimization and is suitable for resource-constrained scenarios.
• Future Direction: The authors note that while MRT improves upon sequential scanning limitations, future work aims to explore inherently spatial state-space models to further overcome the limitations of processing visual text with sequential models.

The work is positioned not as a replacement for all existing STR methods, but as a highly efficient alternative that rivals state-of-the-art performance with a significantly reduced parameter count.