Embedding Rotation Invariance for Provable Multi-Oriented Scene Text Recognition

Technical Summary: Embedding Rotation Invariance for Provable Multi-Oriented Scene Text Recognition

Problem Statement
Scene Text Recognition (STR) faces significant challenges when dealing with multi-oriented text, a ubiquitous occurrence in real-world scenarios where text appears at various angles (e.g., 0°, 90°, 180°, 270°). Existing approaches typically rely on explicit orientation estimation or image rectification (e.g., ASTER, ESIR) to align text horizontally before recognition. These methods suffer from three primary limitations:

1. Lack of Theoretical Guarantees: They depend on the model correctly perceiving orientation; errors in this perception lead to recognition failures.
2. Increased Computational Cost: Additional networks for orientation estimation or rectification modules add inference overhead.
3. Data Reliance: They often require extensive rotation-based data augmentation, forcing the model to learn separate features for different orientations, which reduces feature utilization efficiency.

Furthermore, existing rotation-equivariant networks designed for classification or low-level vision tasks are not directly suitable for STR, which requires variable-length, multi-label sequence generation and the preservation of fine-grained character details.

Methodology
The authors propose RISTER (Rotation-Invariant Scene TExt Recognition), an end-to-end framework that embeds rotation invariance directly into the network architecture rather than relying on post-processing or data augmentation. The system follows an encoder–decoder paradigm with two core components:

1. Rotation-Equivariant Local–Global Extraction (RELG) Encoder:

   • Objective: To extract visual features that transform predictably (equivariantly) under image rotation while preserving fine-grained details and modeling inter-character dependencies.
   • Architecture: RELG integrates deep equivariant convolutions with self-attention. It utilizes F-Conv (Fourier-based group-equivariant convolution) within local blocks to perform rotation-equivariant feature extraction via progressive downsampling. This preserves geometric details often lost in shallow tokenization.
   • Global Modeling: Global blocks employ self-attention mechanisms, which are inherently equivariant to rotations, to capture inter-character dependencies and positional information across the feature map.

2. Rotation-Invariant Text Decoder (RITD):

   • Theoretical Foundation: The authors provide a theoretical proof that the cross-attention mechanism possesses a rotation-invariant property when operating on 2D feature inputs. Specifically, if the query tokens ($Q$) remain fixed and the source tokens (visual features $F$) undergo a spatial rotation ($R_\theta$), the resulting attended features remain identical ($\mathcal{C}(Q, R_\theta(F)) \equiv \mathcal{C}(Q, F)$).
   • Implementation: RITD leverages this property by using an autoregressive decoding scheme where the query tokens are derived from previously decoded characters (via fixed query processing) and remain invariant under image rotation. The visual features serve as the source for cross-attention. This design ensures that the mapping from visual features to the output text space is rotation-invariant, producing consistent predictions regardless of input orientation.

Key Contributions

• Theoretical Proof: The paper establishes a theoretical proof for the rotation-invariant property of cross-attention, demonstrating that it defines a mapping over unordered feature sets insensitive to geometric structures.
• Architectural Design:
  • Proposes RITD, a decoder that achieves rotation-invariant variable-length decoding by adhering to the derived design principles (using cross-attention with invariant queries).
  • Proposes RELG, a hierarchical encoder that combines deep equivariant convolutions with self-attention to balance rotation equivariance, fine-grained detail preservation, and inter-character dependency modeling.
• End-to-End Invariance: By integrating RELG and RITD, RISTER achieves full rotation invariance without additional inference computation or reliance on data-driven orientation correction.

Experimental Results
Experiments were conducted on 14 English benchmarks, including standard datasets (IC13, SVT, etc.) and challenging multi-oriented datasets (Union14M-Benchmarks, ASOT).

• Performance: RISTER achieves state-of-the-art (SOTA) performance on both standard and multi-oriented benchmarks. On the general multi-oriented dataset, it surpasses the second-best model by 4.0% in accuracy.
• Robustness: The model demonstrates consistent recognition accuracy across 0°, 90°, 180°, and 270° orientations, unlike baseline models which suffer significant accuracy drops when input images are rotated.
• Efficiency: Despite using group-equivariant convolutions, RISTER incurs no additional inference cost compared to standard convolutions because parameters are fixed via cyclic shifting after training. Its FLOPs, parameter count, and FPS are comparable to existing AR-based STR methods (e.g., NRTR).
• Ablation Studies: Removing either the Rot-E encoder or Rot-I decoder breaks the end-to-end invariance, confirming the necessity of both components. Furthermore, RISTER outperforms models trained with random rotation augmentation, suggesting that architectural invariance is a more principled approach than data augmentation.

Significance
The paper claims that RISTER provides a provably rotation-invariant solution to multi-oriented STR. Its significance lies in addressing the limitations of prior rotation-aware methods by:

1. Eliminating the need for explicit orientation estimation and rectification modules, thereby reducing computational overhead.
2. Removing the heavy reliance on rotation-based data augmentation, leading to more efficient data utilization.
3. Providing theoretical guarantees for rotation invariance, which enhances model robustness across diverse and challenging real-world scenarios.

The authors conclude that integrating rotation invariance into the model architecture is a superior strategy for multi-oriented recognition compared to data-driven strategies, setting a new benchmark for STR performance. Future work is planned to extend this framework to non-square input settings for tasks like mathematical expression recognition.