💬 NLP

Test-Time Training for Modality Order Consistency in Vision-Language Models

This paper identifies a consistent performance gap in vision-language models caused by the order of image and text inputs, and proposes a test-time training method that not only closes this modality-order gap but also improves overall accuracy by aligning internal representations across different prompt sequences.

Original authors: Aditi Gupta, Yossi Gandelsman

Published 2026-07-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditi Gupta, Yossi Gandelsman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to see and speak at the same time. This is the world of Vision-Language Models (VLMs), a branch of artificial intelligence where computers learn to look at pictures and answer questions about them, or describe what they see. Think of these models as super-smart students who have read every book and seen every photo on the internet. But just like real students, they can get confused by how you ask a question. In the world of AI, the "order" of information matters. If you show a picture first and then ask a question, the robot might think differently than if you ask the question first and then show the picture. This paper investigates a strange glitch: even though the picture and the question are exactly the same, swapping their order changes the robot's answer. It's like if a human got a math problem wrong because you wrote the numbers down in a different order, even though the math itself didn't change. The researchers wanted to know why this happens and if they could teach the robot to stop caring about the order, making it smarter and more consistent.

The authors of this paper discovered that modern AI models are surprisingly sensitive to this "modality order." They tested three different smart models on three different sets of questions and found a consistent pattern: when the picture was shown first, the models got the right answer much more often. When the question came first, the models stumbled, sometimes missing the correct answer by a huge margin—up to 26 percentage points on harder tests. This is a big deal because the information is identical; only the sequence changed. It's as if the robot's brain has a "preferred lane" for processing information, and forcing it into the other lane causes a traffic jam.

To fix this, the researchers invented a clever trick called "Test-Time Training." Instead of retraining the whole robot from scratch (which takes forever), they taught it to learn on the fly, right when it's being asked a question. Here is how it works: for every single question, they ask the robot the same thing twice—once with the picture first, and once with the question first. They noticed that the "picture-first" version usually got it right, so they used that answer as a "teacher." Then, they gently nudged the "question-first" version to think more like the teacher. They did this by taking a few tiny steps of learning just for that specific question, using a mathematical rule that says, "Hey, your answer looks a bit different from the teacher's; let's adjust your brain just a little bit to match."

The results were surprisingly good. This simple nudge didn't just fix the "question-first" mistakes; it actually made the "picture-first" answers slightly better too. It's like if you practiced a song with your left hand, and suddenly your right hand got better at playing it too. The models became more consistent, closing the gap between the two ways of asking questions. In some cases, the "question-first" accuracy jumped from a low 35% to a high 61%, almost catching up to the "picture-first" performance.

But the researchers didn't stop at just fixing the problem; they wanted to know where inside the robot's brain this confusion happened. They used a technique called "activation patching," which is like swapping out a specific gear in a clock to see if it fixes the time. They found that the confusion wasn't happening everywhere in the model. Instead, it was concentrated in a very specific, narrow middle section of the robot's layers. If they swapped the "question-first" brain signals with the "picture-first" signals in this middle zone, the robot suddenly got the right answer. This suggests that the error isn't a general lack of intelligence, but a specific "circuit-level" failure in the middle of the processing chain.

Furthermore, they discovered that the "Test-Time Training" worked best when it focused its learning on these same middle layers. It's as if they found the exact spot where the robot was getting confused and applied a targeted repair. Interestingly, they found that treating both orders as equally reliable (trying to make them meet in the middle) didn't work as well. The robot really needed a clear direction: the "picture-first" version was the reliable teacher, and the "question-first" version needed to learn from it.

In the end, this paper shows that AI models can have weird, specific weaknesses that have nothing to do with the actual task. By spotting these weaknesses and using a smart, one-way learning trick, the researchers were able to make the models more robust. They proved that you can fix these glitches without needing new data or retraining the whole system, just by giving the model a quick, targeted nudge at the moment it needs it. It's a reminder that even super-smart AI can get tripped up by the order of things, but with a little bit of the right kind of help, it can learn to pay attention to the right things.

Technical Summary: Test-Time Training for Modality Order Consistency in Vision-Language Models

Problem Identification
This paper identifies a systematic and repeatable failure mode in modern Vision-Language Models (VLMs): sensitivity to the order in which modalities are presented. Despite the semantic content (image, question, and answer choices) remaining identical, swapping the position of the image and the text prompt leads to substantially different predictions. Across three models (InternVL2, Qwen2.5-VL, Qwen3-VL) and three benchmarks (MMStar, RealWorldQA, AI2D), "image-first" prompting consistently outperforms "question-first" prompting. The accuracy gap ranges from 6 to 26 percentage points, indicating that the relative ordering of modalities induces structured differences in internal computation, even though the task semantics are unchanged.

Methodology
The authors propose a label-free, asymmetric Test-Time Training (TTT) method to repair this inconsistency without requiring external supervision or task-specific fine-tuning.

Asymmetric Adaptation Objective: The method treats the stronger "image-first" prediction as a teacher signal and the weaker "question-first" prediction as the student. For each test instance, the model performs a small number of gradient steps ( $K$ ) to align the question-first output distribution toward the image-first distribution.
Moving Teacher: Crucially, the image-first teacher is not fixed; it is recomputed at each gradient step using the updated model weights, though gradients are detached from the teacher branch. This "moving teacher" approach allows the adaptation to bootstrap both orderings toward mutual consistency.
Mechanism-Guided Scope: While the method can theoretically update all parameters, the authors restrict gradient updates to specific windows of transformer layers. Based on mechanistic analysis, they find that adaptation in mid-to-late network layers yields the best trade-off between recovering question-first accuracy and preserving image-first performance.
Loss Function: The alignment is driven by an asymmetric Kullback-Leibler (KL) divergence loss, $D_{KL}(\bar{p}_{IF} || p_{QF})$ , where $\bar{p}_{IF}$ is the detached teacher distribution.

Key Contributions

Identification of Circuit-Level Failure: The paper establishes modality-order sensitivity as a repeatable failure mode in VLMs, where identical image-question pairs produce divergent predictions based solely on prompt ordering.
Causal Localization via Activation Patching: Using activation patching, the authors demonstrate that this failure is not diffuse but causally concentrated in a narrow mid-network band (specifically layers 18–20 in InternVL2-8B). Replacing question-first hidden states with image-first states in early or late layers has minimal effect, whereas patching in this specific mid-layer window sharply restores accuracy.
Directional Repair Mechanism: The authors propose and validate a mechanism-guided, asymmetric TTT procedure. By using the stronger image-first branch as a teacher to repair the weaker question-first branch, the method closes the accuracy gap while largely preserving (and slightly improving) the stronger ordering.

Results

Gap Closure: The TTT method substantially closes the question-first accuracy gap across all evaluated settings. For example, on the InternVL2-8B model with the MMStar dataset, question-first accuracy improved from 35.5% to 61.5% (a +26.1 point gain), nearly matching the image-first baseline of 61.6%.
Preservation and Improvement of Stronger Ordering: Contrary to the expectation that adapting the weaker branch might degrade the stronger one, the image-first accuracy either remained unchanged or improved slightly (by 0.0 to 1.2 percentage points) in most settings.
Ablation Insights:
- Asymmetry is Essential: Symmetric consistency objectives (treating both orders as equally reliable) recovered significantly less of the gap and slightly degraded image-first performance.
- Moving Teacher is Critical: A frozen teacher (computed once from the pretrained model) resulted in much lower question-first recovery (43.9% vs. 61.5% with a moving teacher), confirming that the bootstrapping effect of updating the teacher is vital.
- Layer Specificity: Adaptation targeting mid-layer windows (e.g., layers 16–20) outperformed adaptation in early or late layers.

Significance and Claims
The paper claims that modality-order sensitivity represents a "circuit-level failure" in VLMs that is causally localizable. The significance of the work lies in demonstrating that simple, asymmetric test-time adaptation can effectively mitigate this failure. The authors argue that because one input variant (image-first) is empirically more reliable, adaptation objectives should preserve this directionality rather than enforcing symmetric consistency. Furthermore, the results suggest that test-time adaptation is most effective when targeted at the specific internal regions where inconsistencies emerge, rather than applying uniform updates across the entire network. The paper concludes that this approach not only repairs the weaker ordering but also produces a more internally consistent model, potentially improving the stronger ordering through the bootstrapping of shared weights.

Technical Summary: Test-Time Training for Modality Order Consistency in Vision-Language Models

More like this