🤖 AI

XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models

This paper introduces XModBench, a large-scale tri-modal benchmark designed to evaluate cross-modal consistency in Omni-Language Models, revealing that even state-of-the-art models like Gemini 2.5 Pro struggle with modality-invariant reasoning, exhibit significant performance disparities across modalities, and show systematic directional imbalances.

Original authors: Xingrui Wang, Jiang Liu, Chao Huang, Xiaodong Yu, Ze Wang, Ximeng Sun, Jialian Wu, Alan Yuille, Emad Barsoum, Zicheng Liu

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xingrui Wang, Jiang Liu, Chao Huang, Xiaodong Yu, Ze Wang, Ximeng Sun, Jialian Wu, Alan Yuille, Emad Barsoum, Zicheng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to understand the world. You show it a picture of a dog, play a recording of a dog barking, and write the words "dog barking." A truly smart robot should realize that all three things mean the exact same thing, no matter how you present them.

This is the core idea behind XModBench, a new "report card" for the next generation of AI models called Omni-Language Models. These are the super-smart AIs that can see, hear, and read all at once.

Here is a simple breakdown of what the paper is about, using some everyday analogies:

1. The Problem: The "Fashion-Forward" Robot

Current AI models are like a student who is great at reading a textbook but terrible at listening to a lecture or looking at a diagram. They might get the right answer if you ask them a question using text, but if you ask the exact same question using audio or a picture, they might get it wrong.

They aren't actually "understanding" the concept of a dog; they are just memorizing that the word "dog" usually appears next to the answer. They are biased toward the way information is delivered.

2. The Solution: The "Swapping Game" (XModBench)

The researchers created a massive test called XModBench to see if these robots are truly smart or just good at guessing based on the format.

Think of XModBench as a magic mirror game.

The Setup: They take one single question (e.g., "What animal is making this sound?").
The Twist: They play the game six different ways:
1. Show a picture, ask for the text answer.
2. Play a sound, ask for the text answer.
3. Show a picture, ask for the sound answer.
4. Play a sound, ask for the picture answer.
5. And so on...

If the AI is truly smart, it should get the answer right every single time, regardless of whether it's looking, listening, or reading. If it gets the answer right when reading but wrong when listening, the test reveals a "glitch" in its brain.

3. The Test Subjects: 61,000 Questions

The researchers didn't just ask a few questions. They built a giant library of 61,320 questions covering five different areas of life:

Perception: Recognizing simple things (Is that a cat or a dog?).
Spatial Reasoning: Understanding where things are (Is the car moving left or right?).
Temporal Reasoning: Understanding time and order (Did the dog bark before or after the car honked?).
Language: Translating or understanding emotions in speech.
External Knowledge: Knowing facts about the world (Who is this singer? What movie is this poster from?).

4. The Results: The "Honest Report Card"

When they ran the top AI models (like Google's Gemini) through this test, the results were eye-opening:

The Good News: The AIs are getting really good at recognizing things when you show them pictures or text. They are like great readers.
The Bad News: They are terrible at listening. When the question came as audio, their performance dropped significantly. It's like a student who aces a written test but fails when the teacher asks them to solve a problem out loud.
The "Directional" Flaw: The AIs also have a weird bias in how they process information. They are great at looking at a picture and guessing the text, but much worse at reading the text and guessing the picture. It's like they are "right-handed" with their brains and struggle when they have to use their "left hand."

5. Why This Matters

This paper is important because it stops us from just looking at the "average score" of an AI. Before, if an AI got 80% right, we thought it was smart. Now, XModBench shows us that maybe it got 95% right on pictures but only 40% right on sounds.

The Big Takeaway:
We are building "Omni" models (models that can do everything), but right now, they are more like "Uni" models (models that are good at only one or two things). XModBench is the tool we need to fix this, forcing developers to teach their AIs to be truly consistent, so they can understand the world whether you show them a photo, play a song, or write a note.

In short: XModBench is the ultimate "lie detector" for AI, ensuring they aren't just memorizing patterns but actually understanding the world behind the pixels, the waves, and the words.

1. Problem Statement

Omni-modal Large Language Models (OLLMs) aim to unify the understanding of text, vision, and audio within a single framework. While existing benchmarks evaluate general cross-modal question-answering (QA) performance, they fail to address a critical gap: cross-modal consistency.

Current research does not sufficiently determine whether OLLMs possess modality-invariant reasoning. Specifically, it is unclear if models converge on the same correct answer when the same semantic content is presented in different forms (e.g., spoken audio vs. written text vs. visual image). Existing benchmarks often fix the context or candidates to a single modality, or they only evaluate specific pairs (e.g., vision-text), leaving the systematic biases and directional imbalances of models unmeasured.

2. Methodology: XModBench Design

The authors introduce XModBench, a large-scale, tri-modal benchmark designed explicitly to measure cross-modal consistency through a balanced, controlled experimental design.

A. Data Construction

Scale: 61,320 multiple-choice question-answer pairs derived from 10,220 unique semantic instances.
Tri-Modal Alignment: Each instance is built from aligned Text-Image-Audio triplets.
Modality Permutation: To test consistency, each semantic instance is instantiated into six distinct modality configurations by permuting the input context and candidate modalities:
1. Audio $\to$ Text
2. Audio $\to$ Vision
3. Text $\to$ Audio
4. Text $\to$ Vision
5. Vision $\to$ Audio
6. Vision $\to$ Text
Data Sources: A combination of re-annotated public datasets (e.g., VGG-Sound, STARSS23), synthetic generation (using TTS and rendering tools), and web-collected data (YouTube, IMDb).
Quality Control: A pipeline involving LLM-based filtering and human-in-the-loop verification ensures high-quality, unambiguous distractors.

B. Task Taxonomy

The benchmark spans five task families and 17 subtasks:

Perception: Recognizing objects/activities (General, Fine-grained, Natural Environment, Instruments, Instrument Composition).
Spatial Reasoning: 2D Arrangement, 3D Localization, 3D Movement.
Temporal Reasoning: Event Order, Temporal Counting, Temporal Calculation.
Linguistic Understanding: Recognition (OCR/ASR), Translation (English-Chinese), Emotion Classification.
External Knowledge: Movie Recognition, Music Genre Classification, Singer Identification.

C. Diagnostic Metrics

XModBench introduces three specific metrics to diagnose model behavior beyond simple accuracy:

Task Competence: Average accuracy across all six modality directions for a specific task.
Modality Disparity ( $\Delta$ ): Measures the performance gap when the same question is posed with different input modalities (e.g., Audio context vs. Vision context). High disparity indicates reliance on modality-specific cues rather than shared semantics.
Directional Imbalance ( $\Delta_{X \leftrightarrow Y}$ ): Measures the asymmetry between inverse settings (e.g., Vision $\to$ Text vs. Text $\to$ Vision). Large gaps suggest the model is biased toward a specific direction (e.g., treating text as the primary output modality).

3. Key Contributions

First Tri-Modal Consistency Benchmark: XModBench is the first benchmark to systematically cover all six cross-modal directions among audio, vision, and text for multiple-choice QA.
Comprehensive Coverage: It spans five diverse domains with 61k+ samples, enabling fine-grained diagnosis of specific weaknesses (e.g., spatial reasoning vs. perception).
New Diagnostic Framework: It defines and quantifies Modality Disparity and Directional Imbalance, providing actionable insights into model robustness and alignment.
Open Resources: The dataset, code, and evaluation tools are released to facilitate future research.

4. Experimental Results

The authors evaluated state-of-the-art closed-source models (Gemini 1.5 Pro, 2.0 Flash, 2.5 Pro) and open-source models (Qwen2.5-Omni, EchoInk-R1, Baichuan Omni 1.5, etc.).

A. Overall Performance

Top Performers: Gemini 2.5 Pro achieved the highest average accuracy (70.6%) and the best balance of stability (Std. 11.7).
Open-Source Gap: While models like Qwen2.5-Omni and EchoInk-R1 perform well on perception and linguistic tasks, they lag significantly behind closed-source models in External Knowledge and Spatial/Temporal reasoning.

B. Key Findings on Consistency

Spatial and Temporal Reasoning are Weak: Even the best model (Gemini 2.5 Pro) struggles with these tasks, achieving <60% accuracy (50.1% for Spatial, 60.8% for Temporal), compared to ~76% for Perception and Linguistic tasks.
Audio is the Weakest Link: Models exhibit significant Modality Disparity. Performance drops substantially when audio is involved compared to text or vision. For Gemini 2.5 Pro, the disparity between Audio and Text ( $\Delta_{T \text{ vs } A}$ ) was -49 points, indicating models are far less capable when reasoning from audio inputs.
Directional Imbalance: Models show systematic asymmetry. For example, accuracy is consistently higher when Text serves as the candidate (output) compared to when it serves as the context. Vision $\to$ Text often outperforms Text $\to$ Vision by significant margins (e.g., 8.8 points for Gemini 2.5 Pro).
Interleaved Data Reduces Imbalance: Models trained on massive interleaved multimodal corpora (like Gemini 2.5 and Qwen2.5-Omni) show smaller directional imbalances compared to models trained on non-interleaved data, suggesting that mixed-modal training helps build symmetric cross-modal bridges.
Post-Training Risks: Fine-tuning on specific datasets (e.g., spoken instructions) can degrade cross-modal alignment, as seen in EchoInk-R1, which showed larger disparities than its base model despite reinforcement learning.

C. Multi-Modal Input (A+V)

When testing a dual-context setting (Audio + Vision $\to$ Text), models showed only modest gains over the best single-modality baseline. This suggests current OLLMs do not yet fully exploit complementary signals or effectively fuse heterogeneous modalities to improve reasoning.

5. Significance and Conclusion

XModBench fundamentally shifts the evaluation of OLLMs from "can it answer?" to "does it reason consistently regardless of input form?"

Diagnostic Value: The benchmark reveals that current OLLMs are not truly modality-invariant. They rely heavily on surface-level patterns specific to certain modalities (especially text) and struggle with audio and complex reasoning tasks.
Guidance for Development: The findings highlight the need for:
- Better interleaved training data to reduce directional bias.
- Enhanced audio representation learning, as audio remains the bottleneck.
- Improved spatial and temporal reasoning capabilities.
- Careful post-training strategies that do not erode pre-trained cross-modal alignment.

The paper concludes that while OLLMs have made impressive strides in perception and language, achieving truly modality-agnostic intelligence requires addressing these systematic inconsistencies, for which XModBench serves as a critical diagnostic tool.