💻 computer science

Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding

This paper proposes Inter-Modal Distance Invariant Position Encoding (DIPE), a mechanism that mitigates visual fading in long-context Multimodal Large Language Models by decoupling position encoding to preserve local intra-modal structure while enforcing consistent inter-modal proximity, thereby maintaining stable visual grounding regardless of text sequence length.

Original authors: Lin Chen, Bolin Ni, Qi Yang, Zili Wang, Kun Ding, Ying Wang, Houwen Peng, Shiming Xiang

Published 2026-03-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lin Chen, Bolin Ni, Qi Yang, Zili Wang, Kun Ding, Ying Wang, Houwen Peng, Shiming Xiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Visual Fading" Effect

Imagine you are sitting in a classroom with a teacher (the AI) and a student (the text you type). On the desk in front of the teacher is a photo (the image input).

Short Class (Short Context): You ask a question about the photo immediately. The teacher looks at the photo, answers correctly, and everyone is happy.
Long Class (Long Context): You ask a question, but before the teacher can answer, you start telling a long story about your weekend, your cat, and what you had for breakfast. You keep talking for thousands of words.

The Problem: In current AI models, as you keep talking, the teacher's eyes slowly drift away from the photo on the desk. By the time you finally ask your question about the photo, the teacher has "forgotten" to look at it. They might guess the answer based on your story, or just make something up. This is called Visual Fading. The AI loses its connection to the image because the text sequence got too long.

Why Does This Happen? (The "Distance Penalty")

Current AI models use a mathematical rule called RoPE (Rotary Position Embedding) to understand where things are in a sentence. Think of this rule like a rubber band.

In a normal sentence, words that are far apart should feel less connected than words that are close together. The rubber band stretches, and the tension (attention) drops. This is good for language; it helps the AI understand that the word "cat" at the start of a paragraph isn't directly related to "dog" at the end.
The Flaw: In Multimodal AI, the "photo" is treated just like another word in the sentence. So, as you type more text, the rubber band stretches between the photo and your question. The math says, "Hey, the photo is now 5,000 words away! It must be very far away, so I shouldn't pay attention to it."

But in reality, the photo isn't "far away." It's sitting right there on the desk the whole time! The AI is tricked by its own math into ignoring the picture.

The Solution: DIPE (The "Anchored Table" Trick)

The authors propose a new method called DIPE (Distance Invariant Position Encoding). They realized we need to treat the "Photo" and the "Text" differently.

Imagine the classroom again, but this time with a special rule:

For Text-to-Text (The Rubber Band): When the teacher talks to themselves (or you talk to the teacher), they still use the rubber band. Words far apart are still considered "far." This keeps the story logical.
For Photo-to-Text (The Anchored Table): When the teacher looks at the photo to answer a question, they ignore the rubber band. Instead, they imagine the photo is anchored to a table right next to them. No matter how many words you speak, the photo stays at "Distance Zero."

How it works technically:

Intra-Modal (Text vs. Text): Uses the standard "Sequential" encoding. Distance matters.
Inter-Modal (Photo vs. Text): Uses an "Anchored" encoding. The photo is assigned a fixed position index that doesn't change, no matter how long the text gets. The AI is told: "The image is always right here, next to your brain. Don't let the length of the conversation push it away."

The Results: Why It Matters

The researchers tested this on 19 different benchmarks (like a giant math test for AI).

The Old Way: As the text got longer (from 0 words to 32,000 words), the AI's ability to answer questions about the image dropped like a stone.
The DIPE Way: The AI's performance stayed flat and stable. Even with 32,000 words of distraction, the AI kept looking at the photo and answering correctly.

The Analogy Summary:

Old AI: Like a person trying to read a map while someone keeps shouting new directions in their ear. Eventually, they forget where the map is.
DIPE AI: Like a person who has the map glued to their forehead. No matter how much noise is happening, they can always see the map clearly.

Key Takeaways

It's Simple: It doesn't require a massive new brain; it just changes how the AI "counts" the distance between the image and the text.
It's Safe: It doesn't break the AI's ability to handle short conversations. It works perfectly for both short and long contexts.
It Fixes the "Fading": It stops the AI from "looking away" from the image just because the conversation got long.

In short, DIPE teaches the AI that an image is always "close," even when the conversation is miles long.

1. Problem Statement: Visual Fading in Long-Context MLLMs

Multimodal Large Language Models (MLLMs) excel at short-context visual question answering but suffer from a critical limitation known as Visual Fading in long-context scenarios.

The Phenomenon: As the text sequence lengthens (e.g., generating hundreds or thousands of tokens), the model's attention to visual tokens diminishes significantly. The model generates text detached from visual constraints, leading to hallucinations or incorrect answers.
The Root Cause: The authors attribute this to the inductive bias of Multimodal Rotary Position Embedding (MRoPE).
- MRoPE treats visual and text tokens in a unified sequential framework.
- During autoregressive generation, the relative distance between the initial visual tokens and the newly generated text tokens grows monotonically.
- RoPE inherently imposes a long-term decay property (distance penalty) to capture linguistic locality. Consequently, as more text is generated, the mathematical attention mechanism suppresses visual signals, forcing the model to "look away" from the image.
- This contradicts human cognition, where a visual reference remains perceptually proximal regardless of how much text is spoken about it.

2. Methodology: Inter-Modal Distance Invariant Position Encoding (DIPE)

The authors propose DIPE, a mechanism that decouples position encoding based on the interaction type between modalities. The core insight is that while intra-modal structures need relative distance decay, inter-modal distances should remain invariant (anchored).

A. Orthogonal Decomposition of Attention

DIPE splits the attention mechanism into two distinct pathways:

Intra-Modal Attention (Text-to-Text, Image-to-Image):
- Strategy: Uses standard Sequential Position Encoding (SPE) (i.e., standard MRoPE).
- Goal: Preserves linguistic locality and the 2D spatial geometry of images.
- Mechanism: Both queries and keys use sequential indices ( $p^{spe}$ ).
Inter-Modal Attention (Text-to-Image, Image-to-Text):
- Strategy: Uses Anchored Position Encoding (APE) for queries and SPE for keys.
- Goal: Enforces a constant perceptual distance between modalities.
- Mechanism:
  - For a text query interacting with an image key, the text query is assigned an anchored position index ( $p^{ape}$ ).
  - This anchor is derived from the first token of the modality segment (e.g., the start of the image or the start of the text block).
  - By broadcasting this anchor to all tokens in the segment acting as queries, the relative distance between the text query and the visual key remains constant, regardless of how many tokens are generated in between.

B. Implementation & Infrastructure Compatibility

Dual Query Views: During the forward pass, the model generates two views of the query vector: one processed with SPE and one with APE.
Split-Kernel Strategy: The attention computation is split into two parallel kernels (Intra-modal and Inter-modal) masked by modality.
LogSumExp Fusion: The outputs of the two kernels are merged using the LogSumExp trick (derived from FlashAttention statistics) to ensure mathematically rigorous normalization without increasing total FLOPs.
KV Cache Compatibility: DIPE is a query-side intervention. It does not alter the Key/Value cache of previous tokens, making it fully compatible with existing KV Cache infrastructure and FlashAttention.
No Extra Parameters: The method requires no additional learnable parameters.

3. Key Contributions

Diagnosis of Visual Fading: Identified that the distance-based decay inherent in MRoPE is the primary cause of visual signal degradation in long contexts, rather than just general attention dilution.
DIPE Mechanism: Introduced a novel position encoding scheme that enforces distance invariance for inter-modal interactions while preserving sequential structure for intra-modal interactions.
Efficient Integration: Demonstrated that this complex geometric re-encoding can be implemented efficiently within standard LLM inference pipelines (FlashAttention, KV Cache) without parameter overhead.
Robustness: Validated that DIPE works across different model scales (0.5B to 3B+), architectures (Qwen series), and complex scenarios (interleaved multi-image contexts).

4. Experimental Results

The authors evaluated DIPE on 19 benchmarks across Perception, Document Understanding, and General VQA.

Long-Context Performance:
- In a controlled "Long-Context VQA" protocol (inserting 1K–32K tokens of distractors between image and query), DIPE significantly mitigated visual fading.
- MRoPE + DIPE achieved a 4.10% average accuracy gain over the MRoPE baseline in long-context scenarios.
- While the baseline MRoPE showed a sharp, continuous decline in accuracy as context length increased, the DIPE-enhanced model maintained a stable performance trajectory.
Short-Context Performance:
- DIPE is non-destructive. On standard short-context benchmarks (where visual fading is not an issue), DIPE-enhanced models maintained strict performance parity with baselines (e.g., MRoPE-I + DIPE matched the vanilla baseline at ~51.4% vs 51.5%).
Layer-Wise Analysis:
- Visual attention analysis revealed that standard MRoPE suppresses visual attention in shallow layers. DIPE successfully restores attention distribution across all layers, ensuring visual constraints are maintained early in the generation process.
Generalization:
- Gains were observed across model sizes, with the smallest model (Qwen2.5-0.5B) showing the largest relative improvement (+8.81%), suggesting lightweight models are more susceptible to visual fading.
- DIPE successfully handled interleaved image-text scenarios and multi-image inputs without disrupting temporal ordering.

5. Significance

This paper addresses a fundamental architectural flaw in current MLLMs regarding long-context visual reasoning. By redefining how position encoding interacts across modalities, DIPE provides a principled solution to the "visual fading" problem.

Theoretical Impact: It challenges the assumption that a single sequential distance metric applies to all token interactions, proposing that inter-modal relationships require a different geometric treatment (invariance vs. decay).
Practical Impact: It enables MLLMs to process long documents, videos, or complex multi-turn conversations while retaining high-fidelity grounding to visual inputs, a crucial step toward reliable autonomous agents and complex multimodal reasoning systems.
Adoption: Since it is parameter-free and compatible with existing inference engines, it offers a low-cost, high-impact upgrade for existing MLLM architectures.