💻 computer science

ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation

This paper introduces AdvSpot, the first grounded adversarial OCR benchmark, and proposes ArmorOCR, a two-stage training framework leveraging observation-transferred self-distillation and group relative policy optimization to significantly enhance large multimodal models' robustness against adversarial visual text while maintaining general OCR performance.

Original authors: Linhan Cao, Siyuan Li, Jun Lan, Liangbo He, Guannan Li, Xiaolei Huang, Jun Jia, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Wei Sun

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linhan Cao, Siyuan Li, Jun Lan, Liangbo He, Guannan Li, Xiaolei Huang, Jun Jia, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Wei Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a growing class of systems known as large multimodal models. These are powerful computers that can see images and read text within them, much like a human looking at a photograph of a street sign or a handwritten note. For years, these systems have become remarkably good at reading clear, standard text. However, a significant gap has emerged between human vision and machine vision. Humans possess an intuitive ability to piece together meaning even when text is hidden, distorted, or blended into a complex background. We can look at a pattern of dots and instantly understand the word they form, or read a sentence written in tiny letters on a crowded shirt. Current artificial intelligence, by contrast, often fails at these specific tasks. It might look at the same image and see only noise, miss the text entirely, or focus on the wrong part of the picture. This vulnerability is not just a minor glitch; it reveals a fundamental weakness in how these machines perceive the visual world, leaving them unable to handle the messy, manipulated text found in the real world.

Researchers have long tried to fix this by teaching computers to "think" harder about images, often by asking them to zoom in, crop, or flip pictures to reveal hidden details. While this works, it is slow and requires extra steps every time the computer looks at a picture. A new study introduces a different approach, one that teaches the model to see these hidden patterns instantly, without needing to change the image first. The team behind this work, led by Linhan Cao and colleagues, first created a new testing ground called AdvSpot. This is a collection of 390 images designed specifically to trap artificial intelligence. The images contain text that is readable to humans but challenging for machines, organized into thirteen different types of visual tricks. These tricks range from text that is rotated or mirrored, to words hidden inside complex patterns, to letters that look like they were drawn by hand or generated by other artificial systems. The dataset is unique because it does not just ask the computer to guess the word; it requires the computer to point to exactly where the text is, read it, and answer a question about it, ensuring the machine is truly seeing the right spot and not just guessing based on context.

To solve the problem of these tricky images, the researchers developed a new training method called ArmorOCR. They realized that while a computer might struggle to see a word in its original form, the same word might be easy to read if the image were slightly stretched, rotated, or resized. Instead of asking the computer to perform these transformations every time it looks at a picture, the researchers taught the model to learn from them during training. They set up a two-step learning process. In the first step, they used a "teacher" model that could see the image in many different, transformed ways. This teacher model, which had the advantage of seeing the text clearly after the image was manipulated, would guide a "student" model that only saw the original, difficult image. The teacher did not just give the answer; it showed the student how to perceive the hidden text by sharing its internal understanding of the transformed views. This allowed the student to internalize the ability to see through the visual tricks without ever needing to see the tricks themselves during the actual test.

The second step of the training focused on refining how the model uses this new ability. The researchers gave the model specific rewards for doing different parts of the job correctly. If the model successfully pointed to the right area of the image, it received a reward for localization. If it read the text correctly, it got a reward for recognition. If it could find all the hidden text in a single image and answer questions about it, it received rewards for those tasks as well. By combining these rewards, the model learned to balance all these skills at once. The result was a system that could look at a single, unmodified image and instantly identify, read, and explain text that was previously invisible to machines.

When tested on their new benchmark, the results were striking. Most existing artificial intelligence models, including some of the largest and most advanced ones available, scored very poorly on the AdvSpot test, often failing to recognize the text more than half the time. The new ArmorOCR system, however, significantly outperformed them all. It improved the ability to recognize adversarial text by a large margin, achieving the highest accuracy across the board. Crucially, the researchers found that this new training did not make the model worse at reading normal, clear text. The system remained just as good at standard reading tasks as it was before, proving that it had learned a new skill without losing its old ones. The study suggests that by teaching models to learn from transformed views during training, we can build artificial intelligence that is far more robust and reliable when facing the complex, manipulated visual world humans navigate every day.

Technical Summary: ArmorOCR

Problem Statement
Large Multimodal Models (LMMs) possess strong general Optical Character Recognition (OCR) capabilities but remain vulnerable to "adversarial OCR" patterns. These are visual text configurations that are easily readable by humans but challenge models to localize and recognize due to systematic weaknesses in visual perception. Existing OCR benchmarks primarily focus on natural or document-style text, while current adversarial OCR evaluations suffer from limited scale, insufficient task diversity, and a lack of region-aware evaluation. Furthermore, recognition-only evaluations fail to distinguish whether a model correctly identified a text region or merely inferred the answer from global context or linguistic priors.

Methodology: ArmorOCR
The authors propose ArmorOCR, a two-stage training framework designed to achieve robust, grounded adversarial OCR perception without requiring additional visual transformations or tools during inference.

Stage 1: Observation-Transferred Self-Distillation (OTSD)
This stage addresses the challenge that adversarial text is often view-dependent (i.e., hidden in the original view but visible in transformed views).
- Setup: A "student" model is conditioned only on the original adversarial image ( $x$ ) and a task prompt ( $q$ ). A "teacher" model shares the same backbone but is conditioned on privileged transformed views ( $T(x)$ ), such as resized, rotated, flipped, stretched, or compressed versions of the image.
- Teacher Guidance: The teacher is further guided by adversarial OCR priors and target text to provide reliable token-level distributions.
- Response-Region-Aware Distillation: Instead of uniform distillation, the framework partitions the model's output into three spans:
  1. Analysis Span: Uses a confidence-gated Jensen–Shannon Divergence (JSD) to selectively transfer reliable intermediate perception signals while suppressing uncertain ones.
  2. Answer Span: Applies forward Kullback-Leibler (KL) divergence to impose strong supervision on the final transcription tokens.
  3. Structural Tokens: Excluded from the loss as they do not contribute to perception.
- Goal: To internalize the perceptual advantages revealed by transformed views into the student model's parameters during training.
Stage 2: Reward-Driven Refinement
This stage refines the grounded OCR perception using Group Relative Policy Optimization (GRPO) with task-conditioned rewards, addressing the need for diverse objectives beyond simple recognition.
- Four Complementary Tasks:
  1. Text-to-Bbox: Localization reward based on Intersection over Union (IoU).
  2. Bbox-to-text: Recognition reward based on normalized Levenshtein similarity.
  3. Full Spotting: Joint region-text prediction reward based on pair-level F1 score.
  4. Grounded VQA: Region-specific question answering reward based on answer inclusion accuracy.
- Goal: To jointly optimize localization, recognition, full spotting, and grounded answering capabilities.

Key Contributions

AdvSpot Benchmark: The authors introduce AdvSpot, the first benchmark for grounded adversarial OCR perception. It contains 390 images with region-level annotations (bounding boxes, transcriptions, category labels) and grounded Visual Question Answering (VQA) pairs. It features a comprehensive taxonomy of 5 primary categories and 13 fine-grained adversarial types (e.g., Spatial Manipulation, Glyph Variations, Visual Encoding) organized by underlying failure mechanisms.
ArmorOCR Framework: A novel two-stage training approach that internalizes transformation-revealed perception via privileged observation transfer (OTSD) and refines grounded perception via task-conditioned GRPO.
Empirical Validation: Extensive experiments demonstrating that ArmorOCR improves adversarial OCR perception while preserving competitive general OCR capabilities.

Results

AdvSpot Performance: On the AdvSpot benchmark, existing open-source and closed-source LMMs struggle, with most achieving overall accuracies below 36%. ArmorOCR achieves the best overall performance, improving upon its base model (Qwen3-VL-8B) by 24.5% in accuracy and achieving the highest average localization IoU (63.3%). Notably, it outperforms the strongest competing model by 65% on the "AIGC Fusion Text" category.
Generalization: ArmorOCR achieves the highest average accuracy on other adversarial benchmarks, AdvOCR and SmuggleBench, outperforming tool-assisted methods like VACoT and Smuggle-CoT, even when those methods rely on larger models or complex prompting.
Capability Preservation: Evaluations on general OCR benchmarks (CCOCR, OCRBench, OCRBench-v2) show that ArmorOCR maintains performance comparable to the base model, indicating that the training strategy enhances robustness without degrading general OCR capabilities.
Ablation Studies: Results confirm that both stages are complementary; Stage 1 is crucial for internalizing perception, while Stage 2 effectively translates this perception into accurate grounded answers. Removing visual transfer or region-aware distillation components leads to performance drops.

Significance
The paper argues that adversarial OCR patterns expose a clear human–AI perception gap. By formulating this as a grounded OCR perception task, the work moves beyond simple recognition to require models to identify relevant text regions and answer region-specific questions. The significance lies in providing a systematic evaluation framework (AdvSpot) that enables fine-grained analysis of adversarial failures and a training methodology (ArmorOCR) that bridges the gap between human and model perception without incurring inference-time latency or complexity. The authors posit that this work facilitates future research into reliable and generalizable visual text perception under challenging conditions.

Technical Summary: ArmorOCR

More like this