💻 computer science

One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting

This paper proposes SPaTS, a reinforcement-optimized framework that improves scene text spotting in MLLMs by routing each text instance through a single anchor visual token selected via SPaSO and refined with directional embedding alignment and patch-enhanced decoding to achieve superior precision over existing methods.

Original authors: Rui Tang, Wentao Yang, Peirong Zhang, Yongxin Shi, Shun Zhang, Huiguo He, Lianwen Jin

Published 2026-07-31

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Rui Tang, Wentao Yang, Peirong Zhang, Yongxin Shi, Shun Zhang, Huiguo He, Lianwen Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read a sign in a busy, chaotic city street. The robot has a super-smart brain that understands language and can write stories, but it's terrible at pointing its finger exactly where a word is on a photo. This is the world of "Scene Text Spotting," a field where computers try to do two things at once: read the words and draw a box around them. For a long time, researchers tried to solve this by showing the robot a whole picture and asking it to guess the location using numbers (like "x is 10, y is 20"). But numbers can be messy and imprecise. More recently, scientists tried a different trick: they let the robot point to little square "patches" of the image, like a grid of tiny tiles, to say, "The word is here." It's like asking the robot to point to a specific tile on a floor to find a hidden message.

However, there's a catch. If you ask the robot to point to many tiles at once to find a word, it often gets confused. It might point to the word, but also accidentally point to the background, the sky, or the word right next to it. It's like trying to find a specific friend in a crowded room by pointing at a whole group of people; you might get the right person, but you're also pointing at strangers, which makes the instruction noisy and unclear. This paper asks a simple, bold question: What if we only let the robot point to one single, perfect tile for each word? Could that be enough? The authors, a team from South China University of Technology, suggest that yes, one patch is actually better than many, provided the robot learns to pick the right one. They built a new system that uses a special kind of "trial and error" learning (called reinforcement learning) to teach the robot how to pick that single best tile, and then uses the rest of the image to draw the perfect outline around the word.

The One-Patch Revolution

The paper introduces a new framework called SPaTS (Single-Patch Text Spotting). Think of the old way of doing things as a messy group project where everyone shouts out ideas at once. The new SPaTS method is like a disciplined team where one person is chosen to speak for the whole group. Instead of the AI trying to aggregate information from a bunch of image patches (little squares of the photo), SPaTS forces the model to select just one "anchor" patch for each word it wants to read.

Why is this better? The authors found that when you use multiple patches, the AI gets "noisy." It starts mixing up the word it's reading with the background or neighboring words, leading to confusion. It's like trying to listen to a single instrument in an orchestra while everyone else is playing loudly; the signal gets lost. By forcing the AI to pick just one, high-quality patch, the signal becomes crystal clear. The model says, "I see this one specific spot," and then uses that spot as a starting point to figure out the rest of the word's shape.

The "Smart Guessing" Game (SPaSO)

Here is the tricky part: How does the AI know which single patch is the "best" one to pick? There is no teacher telling it, "Pick patch number 42." The authors realized that this is a perfect job for Reinforcement Learning, which is like training a dog with treats.

They created a system called SPaSO (Single-Patch Selective Optimization). Imagine the AI is playing a game where it has to guess which tile contains the word.

The Guess: The AI tries to pick a patch.
The Reward: If the patch it picked helps it read the word correctly and draw the box accurately, it gets a "treat" (a reward score). If it picks a bad patch and fails, it gets no treat.
The Learning: Over time, the AI learns that picking certain types of patches leads to treats, and it gets really good at picking the right one without needing a human to show it the answer every time.

The authors designed two specific "treats" for this game. One reward checks if the final result (the text and the box) is good. The other reward checks if the AI at least considered the right patch in its top choices. This dual-reward system ensures the AI doesn't just get lucky once; it learns to consistently find the best evidence.

The Secret Sauce: Direction and Shape

To make this one-patch idea work perfectly, the team added two clever upgrades to the AI's brain:

Directional Embedding Alignment (DEA): Imagine the AI's memory of a patch is like a flashlight beam. Sometimes, the beam is just very bright (high energy) but pointing in the wrong direction. The authors realized the AI was getting distracted by how "bright" a patch was rather than what it actually was. They built a filter that separates the "brightness" (magnitude) from the "direction" (what the patch actually represents). This forces the AI to focus on where the information is pointing, not just how loud it is shouting. It's like telling the robot, "Ignore how bright the light is; look at where the beam is pointing."
Patch-Enhanced Decoding (PED): Once the AI picks its single "anchor" patch, it still needs to know the full shape of the word, which might be curved or long. The single patch is just a starting point. The authors built a decoder that takes that single patch and mixes it with the AI's understanding of language. It's like taking a single clue from a treasure map and combining it with your knowledge of the terrain to draw the full path. This allows the AI to look at the entire image again, using the single patch as a guide, to draw a precise, curved line around the text.

The Results: Less is More

The team tested their new system on several challenging datasets filled with curved, tilted, and crowded text (like signs in a busy market). The results were impressive. Their model, which uses only 2 billion or 4 billion parameters (a relatively small size for AI), beat massive, closed-source models that are much larger and more expensive.

In fact, on some tests, their "one patch" method was significantly better than methods that tried to use many patches. For example, on a dataset called CTW1500, their method achieved an F-measure (a score for accuracy) of 81.2%, while other methods struggled to reach 70%. The paper suggests that by cutting out the noise of multiple patches and focusing on one perfect anchor, the AI becomes much more precise and less confused.

Why It Matters

This paper argues that in the world of AI reading text, less is actually more. By rejecting the idea that "more patches equal better understanding," the authors show that a focused, single-point approach, guided by smart trial-and-error learning, can solve complex problems better than brute force. It's a reminder that sometimes, to find the answer, you don't need to look at everything at once; you just need to know exactly where to look. The authors admit that their method requires some extra training time to learn the "game" of picking patches, but the payoff is a system that is not only more accurate but also more efficient and easier to understand.

Technical Summary: One Patch Is Enough (SPaTS)

1. Problem Statement

Scene text spotting requires the simultaneous recognition of text and precise spatial localization of each transcription to its corresponding image region. While Multimodal Large Language Models (MLLMs) offer strong semantic reasoning, they often lack the fine-grained spatial control necessary for accurate localization. Existing Visual Token Grounding (VTG) approaches attempt to bridge this gap by grounding language generation on visual evidence. However, current paradigms typically rely on multi-patch representations, where multiple image patches are aggregated to represent a single text instance.

The authors argue that for scene text—which is often small, dense, and embedded in complex backgrounds—aggregating multiple patches introduces significant redundant noise, localization ambiguity, and coupled errors. Neighboring patches often contain irrelevant background or adjacent words, leading to "mislocalization" and "ambiguity" that degrade both recognition and geometric precision. Furthermore, identifying the single most informative patch for a text instance is a latent attribute that lacks explicit ranking labels, making it difficult for standard supervised learning to optimize effectively.

2. Methodology

The paper proposes Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token and recovers geometry via full-image refinement. The methodology consists of three core components:

A. Single-Patch Selective Optimization (SPaSO)

To address the lack of oracle labels for the "optimal" patch, the authors reformulate patch selection as a discrete decision problem optimized via Reinforcement Learning (RL).

Framework: Built upon the Group Relative Policy Optimization (GRPO) framework, SPaSO treats the selection of a visual patch index as the policy action.
Dual Reward Mechanism:
1. Patch Generation Reward ( $r_{gen}$ ): Evaluates the final spotting quality (combining text transcription accuracy via edit distance and geometric precision via IoU). This forces the policy to select patches that yield high-fidelity transcriptions and precise boundaries.
2. Patch Dominance Reward ( $r_{dom}$ ): A dense reward that encourages the model to keep ground-truth candidate patches within the top- $k$ predictions. This provides optimization signals even when the sparse generation reward is low.
Joint Optimization: The RL objective for discrete selection is trained jointly with continuous geometry supervision (bounding boxes, Bézier curves, points), allowing the model to learn which patch to select while the decoder learns how to regress its geometry.

B. Directional Embedding Alignment (DEA)

To stabilize the selection process and prevent the model from biasing toward high-energy (high-norm) patches rather than semantically relevant ones, the authors introduce DEA.

Mechanism: DEA decouples the feature magnitude and direction of visual prototypes. The prototype $p_i$ is parameterized as a learnable scale $\gamma_i$ multiplied by the normalized direction $\hat{v}_i$ of the original patch feature.
Training Strategy: The scale $\gamma_i$ is warmed up and then frozen, forcing the model to focus on directional semantic alignment (cosine similarity) rather than feature magnitude during the critical selection phase. This suppresses unstable norm bias in patch competition.

C. Patch-Enhanced Decoding (PED)

Recognizing that a single patch alone may not contain sufficient information for precise boundary regression, the authors propose PED to alleviate the information bottleneck.

Mechanism: PED fuses the selected visual patch prototype ( $p_{at}$ ) with the language hidden state ( $h_t$ ) to form an object query.
Process: This fused query is used in a cross-attention decoder to read from the full-image feature map. This allows the model to leverage rich visual-semantic cues from the entire image to recover precise geometric boundaries (boxes, curves, points) without reintroducing the noise of multi-patch aggregation.

3. Key Contributions

SPaTS Framework: A novel single-anchor visual-token paradigm that replaces noisy multi-patch routing with a single anchor token per instance, followed by full-image geometric refinement. This formulation yields a more compact routing signal and tighter coupling between predicted geometry and target regions.
SPaSO: A reinforcement learning framework that optimizes discrete visual-token selection without oracle optimal labels, utilizing patch-level rewards derived from final spotting outcomes.
Architectural Innovations (DEA & PED):
- DEA: Decouples patch magnitude and direction to ensure robust representation and directional alignment with language states.
- PED: Combines routed local evidence with language semantics to recover precise boundaries, avoiding the information bottleneck of single-patch decoding while preventing multi-patch noise.

4. Experimental Results

The authors evaluated SPaTS on three benchmarks: Total-Text, SCUT-CTW1500, and ICDAR 2015.

State-of-the-Art Performance: SPaTS (in 2B and 4B parameter variants) consistently outperforms both frontier closed-source MLLMs (e.g., GPT-5.4, Claude-Sonnet-4.6, Gemini-3.1-Flash) and specialized OCR models (e.g., HunyuanOCR, PaddleOCR-VL).
Efficiency: Despite using significantly fewer parameters than massive open-source MoE models or closed-source systems, SPaTS achieves superior results, highlighting high parameter efficiency in vision-language alignment.
Ablation Studies:
- Single-Patch vs. Multi-Patch: Single-patch selection significantly outperforms multi-patch and coordinate-text formulations, confirming that multiple patches introduce selection ambiguity.
- RL vs. SFT: The SPaSO (RL) stage provides a substantial boost over standard Supervised Fine-Tuning (SFT), particularly in end-to-end spotting metrics, validating the efficacy of optimizing selection logic via rewards.
- Component Impact: Removing DEA leads to feature drift dominated by norms; removing PED causes a collapse in accuracy, confirming the necessity of fusing language priors with explicit visual signals.

5. Significance and Claims

The paper claims that SPaTS offers a novel visual grounding paradigm for high-fidelity text spotting in complex scenarios. Its primary significance lies in:

Rethinking Granularity: Demonstrating that "one patch is enough" when combined with full-image refinement, challenging the prevailing assumption that multiple patches are necessary for robust grounding.
Bridging the Gap: Successfully bridging the semantic reasoning capacity of large multimodal models with the fine-grained geometric rigor required for OCR, without relying on expensive oracle patch annotations.
Robustness: Providing a solution that handles curved, dense, and occluded text more effectively than traditional methods, as evidenced by qualitative results on difficult datasets like CTW1500 and ICDAR 2015.

The authors acknowledge limitations, noting that the RL stage introduces training overhead and that freezing the magnitude branch in DEA may reduce flexibility under extreme illumination changes. However, they position SPaTS as a significant step toward more precise and efficient scene text spotting.