💻 computer science

Scaling State-Space Models from Lines to Paragraphs: An Ablation of Mamba-based OCR

This paper investigates the scalability of Mamba-based OCR from short lines to full paragraphs, revealing that while State-Space Models offer significant speed advantages over Transformers on synthetic data, they currently underperform on real handwriting due to data scarcity rather than inherent architectural limitations.

Original authors: Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas, Clément Chatelain, Thierry Paquet

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas, Clément Chatelain, Thierry Paquet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a robot that can read handwritten and printed documents, from a single line of text to a full page of a novel. For a long time, the best robots used a "Transformer" brain. This brain is incredibly smart and accurate, but it has a major flaw: it gets slower and slower the longer the text gets. Reading a whole paragraph takes it much longer than reading a single word because it has to constantly re-read everything it just saw to understand the new words. It's like trying to solve a puzzle by looking at every single piece you've already placed every time you add a new one.

This paper introduces a new type of robot brain called Mamba (based on State-Space Models). The big promise of Mamba is that it doesn't re-read the past. Instead, it keeps a "mental summary" of what it has seen so far. This means it can read a whole paragraph just as fast as it reads a single line.

The researchers wanted to know: Is this new Mamba brain actually ready to replace the old Transformer brain for reading long documents?

Here is what they found, broken down simply:

1. The "Perfect World" Test (Synthetic Data)

First, the researchers tested the robot in a "perfect world." They fed it clean, computer-generated text (like Wikipedia articles) that looked perfect, with no smudges or weird handwriting.

The Result: Both the old Transformer and the new Mamba were incredibly accurate (almost perfect).
The Speed: Mamba was the clear winner. It was 1.4 to 4.5 times faster than the Transformer. The longer the text got, the bigger the speed gap became.
The Lesson: In a clean environment, Mamba is a fantastic, lightning-fast alternative that doesn't sacrifice accuracy.

2. The "Real World" Test (Handwriting)

Next, they tested the robot on messy, real-world data: actual handwritten notes from the IAM database. This is where things got tricky.

The Result: Mamba struggled significantly. On handwritten lines, it made many more mistakes than the Transformer. On full handwritten paragraphs, the gap was huge: Mamba was nearly three times worse at getting the words right.
The Analogy: Imagine the Transformer is a student who can look back at their textbook (the image) whenever they get stuck. Mamba is a student who has to memorize the entire textbook in their head before they start writing the essay. If the textbook is messy (handwriting) and the essay is long (paragraphs), the student who relies on memory (Mamba) gets overwhelmed and makes mistakes.

3. The "Why" (The Data Hunger)

The researchers dug deep to find out why Mamba failed on handwriting. They discovered it wasn't necessarily because Mamba's brain was "dumb." It was because Mamba is extremely hungry for data.

The Experiment: When they tried to train Mamba on a small amount of data (like the 8,000 paragraphs available in some datasets), the robot just memorized the answers instead of learning the rules. It failed to learn how to generalize.
The Fix: When they gave Mamba 1 million examples to train on, it suddenly started working perfectly, even on long sequences.
The Conclusion: The problem with handwriting wasn't that Mamba can't do it; it's that we haven't given it enough practice material yet. The Transformer is better at learning from small amounts of data, while Mamba needs a massive library of examples to work its magic.

4. The Final Verdict

The paper concludes with a clear guide on when to use which robot:

Use Mamba for Printed Text: If you are digitizing millions of printed historical newspapers or books, Mamba is the best choice. It is fast, accurate, and saves a lot of computing power.
Be Careful with Handwriting: If you are trying to read messy handwritten notes, Mamba currently lags behind the older Transformer models, especially if you don't have a massive amount of training data.
The Future: To make Mamba work well for handwriting, we need to either feed it much more data (like 1 million examples) or build a "hybrid" robot that combines Mamba's speed with the Transformer's ability to learn from less data.

In short: Mamba is a speed demon that works perfectly in clean, printed environments but currently needs a massive library of practice material to handle the messiness of real handwriting. It's not broken; it just needs more training.

Technical Summary: Scaling State-Space Models from Lines to Paragraphs

Problem Statement
End-to-end Optical Character Recognition (OCR) for historical documents increasingly relies on autoregressive sequence models. While Transformer-based architectures dominate current research, their quadratic attention complexity ( $O(n^2)$ ) limits efficient transcription of long, paragraph-level text. State-Space Models (SSMs), specifically Mamba, offer linear-time decoding and have demonstrated accuracy parity with Transformers on printed historical lines. However, three critical gaps remain: the optimal architectural configuration for paragraph-level Mamba-OCR is unverified; the scaling behavior of Mamba-OCR from short lines to full paragraphs is uncharacterized under controlled conditions; and the generalization of SSM-based OCR from printed text to handwriting remains unexplored.

Methodology
The authors build upon a Mamba-based OCR architecture (Mamba-AR) comprising a shared CNN visual encoder, a bidirectional Mamba multimodal connector, and an autoregressive Mamba decoder. To address the research gaps, the study employs a three-pronged methodology:

Hyperparameter Exploration: A systematic ablation study was conducted on four core hyperparameters: decoder depth ( $L$ ), state dimension ( $N$ ), expansion factor ( $E$ ), and connector depth ($MC$). Experiments utilized synthetic paragraphs derived from English Wikipedia, ranging from 100 to 1,000 characters (1 to 10 lines).
Controlled Scaling Study: Mamba-OCR was compared against a Transformer baseline (DAN) trained under an identical protocol on the same synthetic data. This isolated the effects of sequence length and architecture from document quality variations.
Real-World Validation: The models were evaluated on real datasets: printed historical lines and paragraphs (BnL dataset) and handwritten lines and paragraphs (IAM Handwriting Database).

A critical methodological observation involved training data volume. Preliminary experiments revealed that the autoregressive Mamba decoder failed to converge on long sequences with a fixed set of 8,000 training paragraphs, exhibiting memorization. Stable convergence required generating approximately 1 million dynamically sampled paragraphs per configuration.

Key Contributions

Hyperparameter Optimization: The study identifies the state dimension ( $N$ ) and expansion factor ( $E$ ) as the dominant levers for long-sequence accuracy. The recommended configuration ($L=4, N=256, E=6, MC=1$) matches the defaults used in prior work but is empirically validated as optimal for paragraph-level synthetic data. Notably, the choice of $N=256$ is bounded by implementation constraints (cache size and parallel-scan throughput) rather than an accuracy plateau.
Scaling Behavior: On clean synthetic data, Mamba-OCR maintains sub-1% Character Error Rate (CER) across all lengths (up to 1,000 characters), performing on par with the Transformer baseline. However, Mamba-OCR demonstrates a significant efficiency advantage, running 1.4 to 4.5 times faster than DAN, with the speedup increasing linearly with sequence length due to $O(n)$ vs. $O(n^2)$ complexity.
Generalization Limits and Data Scarcity: On real handwritten data (IAM), Mamba-OCR underperforms significantly compared to Transformers and RetNet-based models (e.g., 10.02% CER vs. 3.51% for DAN on IAM paragraphs). The authors attribute this gap primarily to data scarcity rather than an intrinsic architectural flaw. The autoregressive decoder requires compressing all visual context into a single recurrent state, a process that demands substantially more training diversity (lexical and syntactic) to generalize effectively than Transformer models which can re-attend to the image.

Results

Synthetic Data: Both models achieved <1% CER. Mamba-OCR was 1.4× faster at 1 line and 4.5× faster at 10 lines compared to DAN.
Printed Text (BnL): Mamba-OCR remained competitive with Transformers (6.07% CER vs. 5.24% for DAN on paragraphs) while offering a 2.05× speedup.
Handwritten Text (IAM):
- Lines: Mamba-OCR achieved 8.21% CER, lagging behind DAN (4.23%) and DRetHTR (2.26%).
- Paragraphs: The performance gap widened significantly, with Mamba-OCR at 10.02% CER versus 3.51% for DAN.
Data Volume Sensitivity: Experiments confirmed that 8,000 fixed paragraphs were insufficient for Mamba-OCR to converge on long sequences, whereas 1 million dynamically sampled paragraphs enabled stable training. The IAM dataset (747 training paragraphs) falls within the data-scarce regime where Mamba-OCR struggles.

Significance
The paper clarifies the practical applicability of SSMs for large-scale document transcription. It establishes that Mamba-OCR is a viable, high-throughput choice for printed document digitization at scale, offering substantial speed gains with comparable accuracy to Transformers. However, for handwritten recognition, particularly at the paragraph level with limited training data, the current autoregressive SSM architecture faces significant challenges due to data hunger. The authors conclude that while SSMs offer compelling efficiency, their deployment on novel domains or handwriting styles requires either substantially larger training corpora (via synthetic pre-training) or architectural augmentations (such as hybrid SSM-attention designs) to overcome the inductive bias gap observed in small-data regimes.

1. The "Perfect World" Test (Synthetic Data)

2. The "Real World" Test (Handwriting)

3. The "Why" (The Data Hunger)

4. The Final Verdict

Technical Summary: Scaling State-Space Models from Lines to Paragraphs

More like this