🤖 AI

MULTITEXTEDIT: Benchmarking Cross-Lingual Degradation in Text-in-Image Editing

The paper introduces MULTITEXTEDIT, a controlled benchmark spanning 12 languages and a novel language fidelity metric, to demonstrate that current text-in-image editing systems suffer from significant cross-lingual degradation, particularly in script accuracy for non-Latin languages, despite maintaining global visual structure.

Original authors: Liwei Cheng, Zirui Song, Shibo Feng, Lunjie Zhou, Yixuan Guan, Dayan Guan

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liwei Cheng, Zirui Song, Shibo Feng, Lunjie Zhou, Yixuan Guan, Dayan Guan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital photo of a poster with the words "SALE" written on it. You want to use an AI tool to change those words to "SOLD" without messing up the background, the colors, or the picture of the shirt behind the text. This is called "text-in-image editing."

While these AI tools are getting very good at doing this in English, the paper MULTITEXTEDIT asks a simple but crucial question: What happens when we ask the AI to do the same thing in other languages?

Here is the breakdown of their findings, explained with everyday analogies:

1. The Problem: The "English-Only" Blind Spot

Think of current AI image editors like a chef who is a master at cooking Italian food but has never tried cooking Thai, Arabic, or Japanese cuisine. If you ask them to swap out an ingredient in a Thai dish, they might keep the plate and the table setting perfect, but they might serve you the wrong spice or get the name of the dish completely wrong.

The authors found that existing tests for these AI tools are almost entirely in English. They often judge the AI based on whether the picture looks good (visual plausibility), ignoring whether the words actually make sense (semantic correctness). It's like grading a student on how neatly they wrote a sentence, even if they spelled every word wrong.

2. The Solution: A New "Language Gym" (The Benchmark)

To fix this, the researchers built a massive training and testing ground called MULTITEXTEDIT.

The Setup: They took 300 base images (like blank posters) and created 12 different versions of each, one for 12 different languages (including English, Spanish, Arabic, Hebrew, Chinese, and even less common ones like Yoruba and Bengali).
The Control: Because every version started from the exact same image, they could isolate the language variable. It's like testing a car engine on a track where the only thing that changes is the fuel type, not the road or the weather.
The Scale: This resulted in 3,600 specific test cases covering 7 different types of edits (like changing font size, color, or replacing words).

3. The New Ruler: Measuring "Script Fidelity"

The researchers realized that standard computer tests (which just count how many pixels match) are terrible at spotting language errors.

The Analogy: Imagine a computer checking a handwritten note. If you write "Mother" but miss the accent mark on the "e" (making it "me" which means "tamarind" in Vietnamese), a pixel-counting robot might say, "Hey, that looks 99% identical!" But a human knows the meaning has completely changed.
The Fix: They created a new metric called Language Fidelity (LSF). They used a smart AI judge to look only at the specific words that were changed, checking for tiny details like missing accent marks, reversed letters (common in Arabic or Hebrew), or mixed-up scripts. This judge was trained to spot these "typos" that humans care about but machines often miss.

4. The Results: The "Global Layout" Trap

When they tested 12 different AI systems (both free and paid) on this new benchmark, they found a consistent pattern:

The "Good News": The AI is great at keeping the background looking nice. If you ask it to change text in a picture of a beach, the sand and ocean stay perfect.
The "Bad News": The AI struggles significantly with the actual words, especially in non-English languages.
- The "Layout vs. Meaning" Mismatch: The AI often preserves the shape of the text box and the style of the font perfectly, but the letters themselves are gibberish, backwards, or missing critical marks.
- The Hardest Languages: The AI stumbled the most on Hebrew and Arabic (which read right-to-left) and languages with complex accent marks.
- The Easiest Languages: It did best on Dutch and Spanish, which are closer to English.

5. The Conclusion

The paper concludes that while AI image editors are getting better at "painting," they are still struggling with "writing" in languages other than English. They can copy the look of a foreign sign perfectly but often fail to get the words right.

The authors argue that to build truly useful tools for the whole world, we need to stop testing AI only in English and start measuring how well it handles the specific quirks of every writing system, from the direction of the letters to the tiny dots above them.

Technical Summary: MULTITEXTEDIT

Problem Statement

Text-in-image editing has emerged as a critical capability for visual content creation, yet current evaluation standards suffer from two primary limitations. First, existing benchmarks are overwhelmingly English-centric, failing to address the multilingual reality of global content creation. Second, current evaluation protocols often conflate visual plausibility with semantic correctness. A model output may appear visually consistent with the background while rendering incorrect words, failing to follow instructions, or distorting script-specific characters (e.g., missing diacritics or reversing text direction). Consequently, there is a lack of controlled methods to measure how state-of-the-art multimodal models degrade across different languages and writing systems.

Methodology

Benchmark Construction (MULTITEXTEDIT)

The authors introduce MULTITEXTEDIT, a controlled benchmark designed to isolate language as the primary variable in text-in-image editing.

Dataset Scale: The benchmark comprises 3,600 instances derived from 300 base images.
Multilingual Expansion: Each base image is expanded into 12 typologically diverse languages: Arabic, Bengali, Chinese, Dutch, English, Hebrew, Japanese, Korean, Russian, Spanish, Vietnamese, and Yoruba.
Controlled Variables: By sharing the same visual base, editing intent, and instruction template across all language variants, the benchmark ensures that performance differences are attributable to the target script and language rather than confounding visual variations.
Scope: The dataset covers 5 visual domains (Art, Event, Fashion, Food, Quotes) and 7 editing operations (Replace, Delete, Insert, Color, Font, Relocate, Scale).
Ground Truth: Each instance includes a human-edited reference image and precise region masks for both source and target text regions, verified through a three-stage human annotation protocol involving native speakers.

Evaluation Framework

The paper proposes a dual-track evaluation framework to address the limitations of single-metric approaches:

Semantic Track (LVM Judge):
- Utilizes a Large Vision-Language Model (LVM) to score five dimensions on a 0–5 scale: Instruction Following (IF), Text Accuracy (TA), Visual Consistency (VC), Layout Preservation (LP), and Language/Script Fidelity (LSF).
- LSF Metric: A novel metric designed to detect script-level errors that coarse text matching misses (e.g., missing diacritics in Vietnamese, reversed Right-to-Left order in Arabic/Hebrew, or mixed scripts).
- Two-Stage Protocol: To ensure accuracy, the LVM first traces the edited target text segment by comparing the source, reference, and model output. It then scores the LSF based only on the traced text, ignoring other image content.
- Validation: The LVM judge achieves a Quadratic Weighted Cohen's $\kappa$ of 0.76 against native-speaker annotators, validating its reliability.
Pixel Track:
- Uses standard metrics (MSE, PSNR, SSIM, LPIPS) to quantify background fidelity.
- Employs a dual-mask design (union of source and target masks, dilated) to isolate non-edited regions, ensuring that pixel-level scores reflect the preservation of the original image content rather than the edited text.

Key Results

The authors evaluated 12 open-source and proprietary systems (including Step1X-Edit, Qwen-Image-Edit, GPT-image-1.5, and Nano Banana 2) on the benchmark.

Cross-Lingual Degradation: Every evaluated model demonstrated pronounced performance degradation when moving from English to non-English languages.
- Severity: The degradation was most severe for Hebrew ( $\Delta$ Sem = 0.856) and Arabic ( $\Delta$ Sem = 0.780), and least severe for Dutch ( $\Delta$ Sem = 0.181) and Spanish ( $\Delta$ Sem = 0.184).
- Error Concentration: The performance drop was concentrated in Text Accuracy ( $\Delta$ TA = 0.630) and Language/Script Fidelity ( $\Delta$ LSF = 0.881). In contrast, models maintained relatively higher performance in coarse structural dimensions like Layout Preservation ( $\Delta$ LP = 0.180) and Visual Consistency.
Semantic-Pixel Mismatch: A recurring phenomenon was observed where models preserved global layout and background fidelity (high pixel scores) while failing to generate linguistically correct or script-faithful text (low semantic scores).
Model Performance: Nano Banana 2 achieved the strongest overall performance, ranking highest in semantic scores and pixel metrics while maintaining the smallest English-to-non-English gap among top-tier models. Among open-source models, Qwen-Image-Edit-2511 and FireRed-Image-Edit-1.1 showed the strongest semantic performance.

Significance and Claims

The paper claims that MULTITEXTEDIT provides the first controlled mechanism to systematically analyze multilingual degradation in text-in-image editing. Its significance lies in:

Isolating Language Variables: By controlling visual context, it attributes performance drops specifically to script and language complexity rather than visual ambiguity.
Revealing Hidden Failure Modes: The LSF metric uncovers critical errors (e.g., diacritic loss, directionality reversal) that standard OCR or pixel metrics fail to detect.
Highlighting the Semantic-Pixel Gap: The results demonstrate that high visual quality does not guarantee semantic correctness, particularly for non-Latin scripts and diacritic-sensitive languages.

The authors conclude that progress in this field requires not only improved generation quality but also more language-aware modeling and evaluation strategies that account for script-specific challenges. They position MULTITEXTEDIT as a necessary testbed for developing more robust and equitable text-in-image editing systems.