💬 NLP

Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning

This paper introduces PM4Bench, a strictly parallel multimodal benchmark that reveals OCR as a primary cause of cross-lingual disparities in LVLMs, and proposes a label-free, OCR-centric reinforcement learning strategy to effectively boost multilingual capabilities and reduce performance gaps.

Original authors: Junyuan Gao, Jiahe Song, Jiang Wu, Runchuan Zhu, Guanlin Shen, Shasha Wang, Xingjian Wei, Haote Yang, Weijia Li, Bin Wang, Lijun Wu, Conghui He

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junyuan Gao, Jiahe Song, Jiang Wu, Runchuan Zhu, Guanlin Shen, Shasha Wang, Xingjian Wei, Haote Yang, Weijia Li, Bin Wang, Lijun Wu, Conghui He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a computer can look at a photograph and understand exactly what it sees, reading the signs, the menus, and the handwritten notes just as a human would. This ability, known as visual language understanding, has become a cornerstone of modern artificial intelligence. These systems are now the eyes for digital assistants, helping them navigate our phones, computers, and the physical world around us. However, a significant problem has persisted: while these machines are brilliant at understanding English, they often stumble when faced with other languages, especially when text is embedded inside an image. It remains unclear whether this gap in performance is due to the machine's inability to learn new languages or simply because the tests used to measure them were unfair, comparing apples to oranges. Researchers have long struggled to build a fair test that isolates the machine's true capability from the quirks of the data itself.

A team of researchers has now built a new, rigorous testing ground called PM4Bench to solve this problem. Instead of using different pictures for different languages, which can introduce hidden biases, they created a set of ten languages where the content is strictly identical. Whether the text is in English, Chinese, Arabic, or Russian, the layout, the background, the answer, and the meaning of the question remain exactly the same; only the language itself changes. This allows for a truly fair comparison, like weighing ten different fruits on the same scale. Furthermore, they designed the test to mimic real-world usage, where a computer agent sees a screen as a single image containing both text and graphics, rather than receiving the text and the picture as separate files. This "vision setting" forces the model to read the text directly from the pixels of the image, just as a human would when looking at a smartphone screen.

When the researchers tested ten different large vision-language models on this new benchmark, they discovered a clear pattern. The models performed significantly worse when the text was embedded in the image compared to when it was provided separately. More importantly, the gap between how well the models understood English versus other languages widened dramatically in this image-based setting. The team investigated the cause by running a controlled experiment: they took the same image-based questions and provided the models with the correct text written out for them, removing the need to read it from the picture. When the models were given the text directly, their performance improved, and the gap between languages shrank. This evidence pointed to a single, shared bottleneck: the ability to recognize text within an image, a skill known as optical character recognition. The study found that a model's success in reading text from a picture was strongly linked to its success in answering questions about that picture, regardless of the language.

To address this bottleneck, the researchers developed a new training method that focuses entirely on improving the model's ability to read text from images. They generated a massive amount of synthetic training data, creating thousands of images with random text in ten different languages, without needing any human to write labels or answers. They then used a reinforcement learning strategy, a type of training where the model learns by trying to maximize a score, to teach the model to recognize this text more accurately. Crucially, they designed the scoring system to reward not just getting the text right, but also maintaining the model's ability to think and reason through complex problems. The result was a new version of the model that showed marked improvement across all tasks. It became better at reading text in images, and this improvement carried over to its ability to answer questions and interact with graphical interfaces in all ten languages. The gains were not limited to the specific test; the model also performed better on other existing benchmarks, suggesting that the improvement was genuine and transferable.

The study concludes that the uneven performance of these powerful AI systems across different languages is often driven by a specific weakness in reading text from images, rather than a fundamental inability to understand the languages themselves. By identifying this specific weakness and training the model to overcome it using a vast, label-free dataset, the researchers demonstrated a path toward more equitable artificial intelligence. Their work suggests that if we want AI agents to work reliably for everyone, regardless of the language they speak, we must first ensure they can see and read the text that surrounds them in the visual world. This approach offers a practical, efficient way to boost the capabilities of these systems, ensuring that the benefits of artificial intelligence are shared more broadly across the global population.

Technical Summary: Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning

1. Problem Statement

Evaluating the multilingual capabilities of Large Vision-Language Models (LVLMs) is currently hindered by two primary methodological flaws in existing benchmarks:

Non-Parallel Corpora: Most benchmarks rely on translated or naturally occurring multilingual data where content, layout, and difficulty vary across languages. This introduces uncontrolled variance, making it impossible to distinguish whether performance gaps stem from fundamental model limitations or dataset inconsistencies.
Text-Image Separation: Traditional benchmarks often process text prompts and images as separate streams. This diverges from realistic deployment scenarios (e.g., AI agents interacting with smartphones or desktop interfaces) where text is embedded within the visual scene, requiring the model to perform "text-in-image" reading.

Consequently, the drivers of cross-lingual performance disparities and the specific causes of performance drops when moving from traditional to "vision" settings remain undiagnosed.

2. Methodology

A. PM4Bench: A Strictly Parallel, Vision-Centric Benchmark

To address evaluation gaps, the authors introduce PM4Bench, the first multimodal, multilingual, multi-task benchmark built on a strictly parallel 10-language corpus.

Strictly Parallel Content: The benchmark covers 10 languages (English, Chinese, Korean, Thai, Vietnamese, Russian, Hungarian, Serbian, Czech, and Arabic). For every task, semantically identical content is rendered with identical layouts, fonts, and backgrounds; only the language changes. This enables "apples-to-apples" cross-lingual comparison.
Vision Setting: Text and queries are "printed" directly onto images, simulating unified visual observations where agents must parse on-screen text without separate text inputs.
Four Diagnostic Tasks:
1. MDUR (Multi-Discipline Understanding and Reasoning): College-level multimodal reasoning (MCQ) derived from MMMU-pro.
2. MIQA (Multi-Image Question Answering): Open-ended QA over multiple images.
3. MSOCR (Multi-Scale OCR Challenge): Probes text-recognition limits using semantically meaningless words rendered in decreasing font sizes (40pt to 2pt).
4. MGUI (Multilingual GUI Grounding): Evaluates coordinate-based grounding on synthetic webpage templates (e.g., flight booking, banking) to test agent interaction capabilities.

B. Diagnosis: OCR as the Bottleneck

Using PM4Bench, the authors evaluated 10 leading LVLMs (including commercial APIs like GPT-5, Gemini, and open-source models like Qwen3-VL).

Findings: The vision setting consistently caused performance drops and sharpened cross-lingual disparities compared to traditional settings.
Hypothesis Validation: The authors identified OCR as the shared bottleneck. Evidence includes:
- Providing ground-truth text alongside vision inputs significantly narrowed performance gaps.
- Strong per-language correlations were found between MSOCR scores and downstream VQA performance.
- Low-resource, visually dense scripts (e.g., Arabic, Thai) consistently lagged behind high-resource scripts.

C. Solution: OCR-Centric GRPO Reinforcement Learning

Motivated by the diagnosis, the authors propose a training strategy to boost multilingual capabilities without expensive task-specific VQA supervision.

Approach: They apply Group Relative Policy Optimization (GRPO) to the Qwen3-VL-8B-Thinking model.
Data: Training utilizes fully synthesized, label-free OCR data. Text is rendered on random backgrounds with varied fonts and sizes, using semantically meaningless noun combinations to prevent semantic bias.
Reward Design: A dual-reward function is employed to improve OCR while preserving general reasoning:
1. OCR Accuracy Reward ( $R_{acc}$ ): Based on normalized Levenshtein distance between predicted and ground-truth text.
2. Format/Length Reward ( $R_{fmt}$ ): Penalizes outputs that are too short, too long, or repetitive, and crucially, prevents the model from abandoning Chain-of-Thought (CoT) reasoning, which pure accuracy optimization tends to collapse.
Outcome: The resulting model, QGO-8B, is trained without any PM4Bench evaluation examples, ensuring the gains reflect genuine generalization.

3. Key Results

Performance Gains: QGO-8B improves average scores ( $S_{avg}$ ) across all four tasks and settings on PM4Bench. Notably, it significantly reduces the coefficient of variation ( $S_{cv}$ ) across languages, particularly for the weakest languages (Arabic and Thai).
Cross-Task Transfer: Despite being trained solely on synthetic OCR data, QGO-8B improves performance on non-OCR tasks (MDUR, MIQA, MGUI), confirming that enhancing OCR skills lifts the entire multimodal reasoning capability.
External Benchmark Transfer: The improvements transfer to external benchmarks:
- CC-OCR: +4.7 NED overall, with gains in both trained and unseen languages.
- OCRBench: Matches the baseline (791 vs 789) with no regression.
- MTVQA: Gains +0.5 on trained languages while preserving performance on unseen languages.
Ablation Studies: The dual-reward design is critical. Training with only OCR accuracy ( $R_{acc}$ ) collapses reasoning performance on complex tasks (MDUR drops from 34.88 to 26.67), while the format reward ( $R_{fmt}$ ) alone offers limited gains. The combination is necessary to maintain reasoning integrity while boosting recognition.

4. Significance and Claims

The paper claims to close the loop between evaluation and improvement for the equitable global deployment of LVLM-driven agents:

Fair Evaluation: PM4Bench provides the first strictly parallel, vision-centric yardstick to isolate model capability from dataset bias.
Bottleneck Identification: It establishes OCR robustness as a primary driver of cross-lingual disparity in vision settings, a factor often overlooked in favor of pure language modeling.
Efficient Mitigation: The proposed OCR-centric GRPO offers a low-cost, label-free pathway to boost multilingual performance. By using synthetic data and a reward-constrained RL approach, it avoids the need for expensive human-annotated VQA datasets while effectively transferring gains to downstream tasks.

The authors maintain modesty regarding the scope, noting that while OCR-centric RL is a high-leverage intervention, genuine cross-lingual capability ultimately relies on base-model improvements, and the synthetic nature of the training data may not fully capture the complexity of real-world deployments.