Technical Summary: Comparative Evaluation of Machine Translation Systems on Images with Text

Problem Statement

The paper addresses Text-Image Machine Translation (TIMT), a task situated at the intersection of Computer Vision (CV) and Natural Language Processing (NLP). The core challenge is translating textual content embedded within images into a target language. While traditional Machine Translation (MT) handles pure text, TIMT must first locate and extract text from visual data (often in natural scenes with complex backgrounds, distortions, and varying lighting) before translating it. The authors aim to evaluate and compare different architectural paradigms for solving this problem: modular pipelines, Multi-Modal Large Language Models (MLLMs), and end-to-end image-to-image models.

Methodology

Dataset

The experiments utilize a synthetic multilingual dataset introduced by Lan et al. (2024), specifically designed for In-Image Machine Translation (IIMT). The dataset consists of 512×512 images generated by rendering parallel text from IWSLT14 (German–English) and IWSLT17 (French–English, Romanian–English). The text is rendered using the Arial font with random variations in position, rotation, and background color to introduce moderate visual noise while maintaining readability. The dataset is split into training, validation, and test sets across four language pairs: De–En, Fr–En, Ro–En, and their reverses.

Architectural Paradigms Evaluated

The study compares three distinct approaches:

1. Modular Pipelines:

   • OCR Stage: Text is extracted using docTR (selected over EasyOCR based on preliminary testing). The pipeline uses the Faster Arbitrarily-Shaped Text Detector (FAST) for detection and a VGG16-BN backbone with an RNN for recognition.
   • Translation Stage: The extracted text is fed into various translation models:
     • Traditional NMT: M2M100-1.2B.
     • LLMs: Instruction-tuned models from the Llama family (3.2-1B, 3.2-3B, 3.1-8B) and EuroLLM (1.7B, 9B). These models require specific prompting strategies (e.g., JSON formatting or direct commands) to ensure clean, translation-only outputs.

2. Multi-Modal Large Language Models (MLLMs):

   • Three configurations of Gemini-2.5 (flash-lite, flash, and pro) were evaluated. These models process the raw image and a text prompt jointly in a single step, performing OCR and translation simultaneously without a separate OCR stage.

3. End-to-End Model:

   • Translatotron-V: An image-to-image model that directly generates translated images. Results for this model were obtained via private communication with the authors and were limited to specific language pairs due to the lack of public checkpoints.

Evaluation Framework

Performance was measured using standard metrics:

• OCR Quality: Character Error Rate (CER) and Word Error Rate (WER).
• Translation Quality: BLEU, chrF, and TER (Translation Error Rate).
• Statistical Significance: Approximate randomization testing (ART) with 10,000 repetitions was used to determine significant differences between systems.

Key Results

OCR Performance

In preliminary evaluations, docTR significantly outperformed EasyOCR across all language pairs, reducing Word Error Rates by up to 13.9 percentage points and Character Error Rates by up to 12.5 points. Consequently, docTR was used for all subsequent modular pipeline experiments.

Translation Performance

The results reveal a consistent performance hierarchy across all language pairs (De–En, En–De, Fr–En, En–Fr):

1. MLLMs (Gemini-2.5): The multi-modal models achieved the best overall performance. Specifically, Gemini-2.5-pro consistently secured the highest BLEU, chrF, and lowest TER scores across all directions. This demonstrates the superiority of joint visual and linguistic processing.
2. Modular Pipelines (LLMs): Large-scale LLMs, particularly EuroLLM-9B and Llama-3.1-8B, significantly outperformed traditional NMT baselines (M2M100) and smaller LLMs. The modular pipeline using EuroLLM-9B often approached or surpassed the performance of the MLLMs in specific metrics, though generally trailing the top Gemini configurations.
3. Traditional NMT: M2M100-1.2B served as a reliable baseline but was generally outperformed by larger LLMs and MLLMs.
4. End-to-End (Translatotron-V): This approach performed significantly worse than both modular and multi-modal approaches. In the English–German direction, it achieved a BLEU score of ~17.9, compared to ~35.6 for Gemini-2.5-pro. In French–English, it performed better than the NMT baseline (BLEU ~34.2) but still lagged behind LLMs and MLLMs.

Significance and Claims

The paper claims that multi-modal reasoning is highly effective for translating text within images. The primary findings are:

• Superiority of MLLMs: Multi-modal models like Gemini-2.5 demonstrate superior flexibility and contextual understanding, achieving the best translation quality by processing visual and textual information jointly.
• Competitiveness of Modular Pipelines: While MLLMs lead, modular pipelines combining state-of-the-art OCR (docTR) with large, specialized LLMs (like EuroLLM) offer a highly competitive alternative that outperforms end-to-end image-to-image models.
• Limitations of End-to-End Approaches: The study suggests that direct image-to-image translation (as attempted by Translatotron-V) remains a challenging task with current technology, yielding significantly lower quality than approaches that separate text extraction and translation or utilize multi-modal reasoning.

The authors conclude that while modular pipelines are robust, the integration of visual understanding and language generation within MLLMs provides a solid foundation for future research in multilingual image translation. The study is limited by its reliance on synthetic data and a narrow set of European language pairs, suggesting that real-world robustness and low-resource language performance require further investigation.