💻 computer science

Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

This paper identifies the "Ghost Anchor" phenomenon, where early-layer English-centric translation renders visual signals functionally invisible in multilingual MLLMs, and proposes the ANCHOR framework with Proactive Visual Anchoring to restore visual-linguistic alignment and significantly improve non-English visual reasoning performance.

Original authors: Yihang Du, Juhao Liang, Zhengzhao Lai, Siyu Li, Yan Hu

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yihang Du, Juhao Liang, Zhengzhao Lai, Siyu Li, Yan Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a computer can look at a photograph and describe it perfectly in any language a person speaks. This is the promise of modern artificial intelligence, specifically a type of system known as a multimodal large language model. These systems are built by combining a powerful text-processing brain with a visual eye, allowing them to understand both words and images simultaneously. For years, researchers have believed that vision acts as a universal bridge. The logic was simple: an image of a giraffe is the same object whether you call it "giraffe," "jirafa," or "zhǎngjǐnglù." Therefore, the visual part of the computer should help it translate concepts across languages, grounding abstract words in concrete reality. However, despite their impressive skills in English, these systems often stumble when asked to reason about images in other languages, failing to live up to the ideal of a truly universal understanding.

A team of researchers set out to understand why this disconnect happens. They suspected the problem was not a lack of data, but rather a timing issue within the computer's internal processing. By peering inside the layers of these models, they discovered a phenomenon they call the "Ghost Anchor." In a perfectly synchronized system, the visual information and the language information would arrive and merge at the same time, allowing the image to guide the translation. Instead, the researchers found that the computer's language center rushes to translate non-English words into an English mental framework almost immediately. Meanwhile, the visual part of the system is still slow to process what it is actually seeing. By the time the language has settled into its English-centric meaning, the visual details are still too vague to offer any help. The image is physically present in the computer's memory, but it is semantically invisible, like a ghost that cannot touch the conversation.

To prove this, the researchers conducted a series of experiments where they replaced the actual images with random static noise that looked like television snow but kept the same size and shape. They found that in the early stages of processing, the computer's translation of the text did not change at all, even when the picture was gone. This confirmed that the visual signal was not influencing the language translation when it mattered most. The computer was effectively translating the text "blindly," relying on its internal English habits rather than the picture in front of it. This delay creates a missed window of opportunity where the image could have anchored the meaning of the words, preventing the system from drifting into errors.

The researchers then proposed a solution called ANCHOR, which stands for a framework designed to fix this timing mismatch. Instead of waiting for the visual information to naturally catch up, they introduced a method to force the computer to pay attention to the image much earlier in its processing chain. They used a separate, highly trained visual system to act as a teacher, showing the main model exactly what the image represents before the language translation begins. This proactive guidance ensures that the visual details are ready and waiting to influence the text as soon as the translation starts. It is akin to ensuring a translator has the dictionary open and the picture on the table before they begin speaking, rather than waiting until they are already halfway through a sentence to realize they need to look at the reference.

When tested on a wide range of benchmarks involving complex reasoning and cultural questions, this new approach showed clear results. The models trained with this method performed significantly better at answering questions about images in languages other than English, including difficult, low-resource languages. Crucially, they did not lose their ability to understand English; in fact, their performance in English remained stable or improved slightly. The study suggests that by synchronizing the arrival of visual and linguistic information, the computer can build a more robust understanding that works across borders. This finding challenges the idea that simply feeding more data is the only path forward, pointing instead to the importance of how different types of information are timed and combined inside the machine. The work offers a clearer path toward artificial intelligence that can truly see and speak the world's languages with equal fluency.

Technical Summary: Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

Problem Statement

Multimodal Large Language Models (MLLMs) exhibit significant performance degradation in non-English visual reasoning tasks, despite possessing strong multilingual capabilities in their text-only backbones. While current literature often attributes this disparity to data scarcity, this paper argues that the root cause lies in the internal representational mechanisms of MLLMs. Specifically, the authors investigate why visual signals fail to act as a universal, language-agnostic bridge for cross-lingual alignment. They hypothesize that a temporal disconnect exists between the evolution of linguistic and visual representations within the Transformer architecture, preventing visual grounding from effectively guiding translation in non-English contexts.

Methodology: Mechanistic Analysis and Intervention

1. Mechanistic Diagnosis: The "Ghost Anchor" Phenomenon

Through rigorous layer-wise analysis of models like LLaVA-1.5, the authors identify a phenomenon termed Modality Asynchrony, which manifests as the Ghost Anchor effect.

Linguistic Convergence: Linguistic representations rapidly converge toward an "English Semantic Manifold" ( $S_{en}$ ) in the early layers of the network (layers 1–5). Non-English tokens are implicitly translated into English-centric representations before visual semantics are fully processed.
Delayed Visual Semanticization: In contrast, visual tokens remain in a "semantically immature" state (encoding low-level sensory features) during these early layers. They only achieve meaningful semantic grounding (decoding into object labels) much deeper in the network (post-layer 20).
The Consequence: Because the linguistic stream aligns with the English manifold before visual semantics are available, the model performs "blind" translation. The visual modality is physically present but exerts negligible causal influence during the critical early alignment window. This was validated via Visual Causal Effect experiments, where replacing visual inputs with matched Gaussian noise resulted in near-zero changes to early-layer translation metrics ( $\Delta RET$ and $\Delta Sim_{en}$ ).

2. Proposed Solution: ANCHOR

To rectify this asynchrony, the authors propose ANCHOR, a training framework centered on Proactive Visual Anchoring (PVA).

Core Mechanism: PVA explicitly supervises the early-layer evolution of visual representations to accelerate their semantic emergence.
Implementation:
- An external Visual Foundation Model (VFM), specifically SigLIP-SO400M/14, serves as a semantic supervisor.
- A lightweight alignment network ( $MLP_{align}$ ) projects the MLLM's intermediate hidden states from early layers ( $L_{early}$ ) into the VFM's feature space.
- The training objective minimizes the negative mean cosine similarity between the MLLM's early visual tokens and the VFM's fine-grained semantic features.
Training Strategy: The framework applies full-parameter fine-tuning (FFT) to the multimodal projector and the first $L_{early}$ (set to 10) transformer layers, while freezing deeper layers and the vision encoder. This ensures visual features become "semantically ready" before the linguistic stream completes its cross-modal alignment.

Key Contributions

Formalization of the Ghost Anchor Phenomenon: The paper systematically identifies and formalizes the temporal misalignment where visual signals are semantically inaccessible during the early alignment phase of multilingual MLLMs.
Mechanistic Evidence: Through layer-wise analysis and causal interventions (noise substitution), the authors demonstrate that visual information has negligible causal influence on the linguistic translation trajectory in early layers, explaining the failure of vision to bridge non-English gaps.
The ANCHOR Framework: A novel training method that uses external VFMs to enforce early visual semantic readiness, effectively synchronizing dual-stream convergence.

Empirical Results

Experiments were conducted on xMMMU (expert domain reasoning), MaXM (cross-lingual transfer), and CVQA (culturally nuanced VQA) across 7B and 13B parameter models (LLaVA-1.5, LLaVA-NeXT, InternVL3).

Performance Gains: ANCHOR consistently outperforms standard baselines (Std. LoRA) and early-layer fine-tuning without anchoring (Early-Layer FFT).
- On xMMMU, ANCHOR improved average accuracy by ~2.2% (7B) and ~2.3% (13B) over Std. LoRA.
- On MaXM, gains were observed across both fine-tuned and zero-shot languages, with the 13B model improving from 32.6% to 36.5% average accuracy.
- On CVQA, ANCHOR mitigated the performance degradation often seen in early-layer adaptation, improving Arabic accuracy from 28.1% (Early-Layer FFT) to 40.1% (ANCHOR) on the 13B model.
Zero-Shot Generalization: The method demonstrated robust generalization to unseen languages, reducing the "alignment tax" typically incurred during supervised fine-tuning.
English Retention: Unlike some alignment methods that degrade native English proficiency, ANCHOR preserved or slightly enhanced English performance on benchmarks like xMMMU.
Mechanistic Validation: Post-training analysis confirmed that ANCHOR successfully increased the Visual Causal Effect ( $\Delta RET$ and $\Delta Sim_{en}$ ) in early layers, proving that visual signals now actively influence the translation process. Case studies showed ANCHOR could override incorrect linguistic priors (e.g., predicting "laptop" based on text) with visually grounded concepts (e.g., "mesa"/table).

Significance and Claims

The paper claims that the failure of vision to serve as a universal bridge in multilingual MLLMs is not merely a data issue but a structural modality asynchrony. By shifting the focus from external data curation to internal representational synchronization, the authors demonstrate that Proactive Visual Anchoring can rectify the "Ghost Anchor" phenomenon.

The significance lies in providing a mechanistic explanation for cross-lingual visual reasoning failures and offering a targeted, computationally efficient intervention that aligns visual and linguistic streams at the precise layer where cross-modal interaction is most critical. The authors position this work as a step toward truly language-agnostic multimodal understanding, where visual evidence actively guides linguistic translation rather than being bypassed by English-centric internal dynamics.