Technical Summary: Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

Problem Statement

While Vision-Language Models (VLMs) have demonstrated strong performance on OCR-based benchmarks and text-rich understanding tasks, their robustness under controlled visual degradation remains insufficiently understood. This gap is particularly critical for OCR reasoning, where visual corruption does not merely blur semantic content but can induce discrete symbolic errors. These errors are subsequently amplified by downstream reasoning tasks, introducing significant uncertainty. Existing benchmarks largely focus on clean images or general object recognition robustness (e.g., ImageNet-C), failing to isolate the specific phenomenon where visual corruption disrupts text extraction and structured reasoning. Furthermore, recent OCR robustness benchmarks primarily evaluate general OCR tasks rather than the complex reasoning required for text-rich images.

Methodology

Benchmark Construction: OCR-Robust

To systematically study this problem, the authors introduce OCR-Robust, a benchmark designed to evaluate OCR reasoning robustness under visual perturbations. The dataset comprises 812 samples divided into two complementary subsets:

• OCR1.0 (482 samples): Covers natural text perception, including documents, scene text, receipts, handwriting, screenshots, and mathematical content.
• OCR2.0 (330 samples): Focuses on structured visual reasoning, targeting charts, geometry diagrams, and tables.

The construction leverages existing high-quality datasets (e.g., DocVQA, ChartVQA, GSM8K) and employs VLM-assisted annotation with human verification to generate reasoning questions. A synthetic generation pipeline is used for the mathematical subset to ensure ground-truth availability.

Perturbation Selection Pipeline

Rather than applying perturbations ad hoc, the authors conducted a pilot study over 18 candidate perturbation types across 6 models. They utilized three metrics to select the most informative perturbations:

1. Mean Relative Drop (MRD): Measures average performance degradation.
2. Separability (SEP): Quantifies the ability to differentiate models with distinct robustness profiles.
3. Monotonicity (MON): Ensures performance degrades consistently as severity increases.

Based on these metrics, 5 representative perturbations were selected, each applied at 3 severity levels, resulting in 15 evaluation conditions:

• Glass Blur
• Motion Blur
• Elastic Deformation
• Color Shift
• Snow

Evaluation Metrics

The authors propose a suite of metrics to evaluate robustness beyond simple clean accuracy:

• Clean Accuracy: Baseline performance on uncorrupted images.
• Relative Corruption Retention (RCR): The average ratio of corrupted accuracy to clean accuracy, normalizing for baseline capability.
• Worst-Case Retention (WCR): The minimum retention ratio across all perturbations, capturing the most severe failure modes.
• Corruption Robustness Index (CRI): A composite metric using the geometric mean of Clean Accuracy, RCR, and WCR to penalize imbalance and prevent high baseline accuracy from masking poor robustness.

Experimental Setup

The study evaluates 18 models, including:

• Closed-source VLMs: GPT-5.2, GPT-5.1, Gemini-3-Flash (with and without thinking mode).
• Open-source VLMs: Qwen3-VL (various sizes and thinking modes), InternVL3.5, Llama-4-Scout, and others.
• OCR + LLM Pipelines: Two-stage systems where PaddleOCR-VL-1.5 extracts text, which is then fed to an LLM for reasoning.

Experiments were conducted using a zero-shot protocol without fine-tuning.

Key Results and Findings

1. Closed-Source Dominance: Closed-source models generally lead in overall robustness. Top-performing closed-source models achieved CRI scores above 0.90 on OCR1.0 and 0.84 on OCR2.0, maintaining near-perfect average retention and strong worst-case resilience.
2. Capability Does Not Guarantee Robustness: Higher clean accuracy does not necessarily imply stronger robustness. "Thinking mode" models achieved the highest clean accuracy but exhibited significant drops in worst-case retention on structured content (e.g., Qwen3-VL-8B-Thinking had a WCR of 0.624 on OCR2.0). Additionally, within the InternVL3.5 family, the 14B model outperformed the 38B model in worst-case retention despite lower clean accuracy.
3. Fragility of Structured Content: Structured visual content (charts, tables, geometry) is substantially more fragile than document-like inputs. The average WCR declined from 0.826 on OCR1.0 to 0.676 on OCR2.0 (an 18% relative decline), with color shift and snow perturbations being primary drivers of this gap.
4. Pipeline Sensitivity: OCR+LLM pipelines are highly sensitive to the quality of the initial OCR extraction. While they showed high retention ratios (RCR), their low clean accuracy resulted in the lowest overall CRI scores.
5. Limitations of Prompt-Induced CoT: Prompt-induced Chain-of-Thought (CoT) improved clean accuracy significantly but did not reliably prevent worst-case failures. In some cases, WCR actually declined for structured content, suggesting that extended reasoning does not inherently mitigate symbolic brittleness under corruption.

Significance and Contributions

The paper makes three primary contributions:

1. A Diverse Benchmark: OCR-Robust provides a controlled diagnostic setting covering the full spectrum from natural text perception to structured visual reasoning, addressing a gap in existing evaluations.
2. A Principled Selection Method: The pilot study and metric-driven selection of perturbations offer a reproducible framework for identifying informative visual corruptions for OCR reasoning.
3. A Robustness Metric Suite: The introduction of RCR, WCR, and CRI provides a more nuanced view of model stability than clean accuracy alone.

The authors conclude that OCR reasoning robustness should be treated as a distinct evaluation problem. Their results suggest that current models, even those with high clean accuracy or advanced reasoning capabilities, remain vulnerable to visual perturbations, particularly when dealing with structured visual content. Future OCR-capable VLMs require more robust reasoning mechanisms that can withstand the amplification of symbolic errors caused by visual degradation.

Limitations

The authors acknowledge several limitations:

• The dataset size (812 samples) and language coverage are moderate.
• The perturbation selection was based on a pilot study with a narrow basis (100 questions, 6 models).
• The evaluation focuses on single perturbations, whereas real-world degradations often co-occur (though a small-scale compound perturbation pilot was conducted).
• The zero-shot protocol may underestimate robustness under optimized prompting.
• The pipeline evaluation is limited to a single OCR engine (PaddleOCR-VL-1.5).
• Closed-source coverage is limited to specific versions of GPT and Gemini due to budget constraints.