💻 computer science

Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models

This paper introduces XTC-Bench, a novel evaluation framework and the Continuous Cross-Task Agreement (CCTA) metric, to diagnose and quantify the semantic consistency between visual understanding and generation in unified multimodal models, revealing that high standalone task performance does not guarantee coherent unified representations.

Original authors: Weixing Wang, Liudvikas Zekas, Anton Hackl, Constantin Alexander Auga, Parisa Shahabinejad, Jona Otholt, Antonio Rueda-Toicen, Gerard de Melo

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weixing Wang, Liudvikas Zekas, Anton Hackl, Constantin Alexander Auga, Parisa Shahabinejad, Jona Otholt, Antonio Rueda-Toicen, Gerard de Melo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented artist who can also act as a museum guide. This artist is a "Unified Multimodal Model" (uMM). Their job is to do two things with the same picture:

Understand it: Look at a photo and describe what's happening (the "Guide" role).
Create it: Read a description and draw a new picture based on it (the "Artist" role).

For a long time, we've tested these artists by checking their Guide skills separately from their Artist skills. We'd ask, "How well can they describe a car?" and "How well can they draw a car?" separately. If they got high scores on both, we assumed they were perfect.

The Problem: The "Split Personality" Issue
The authors of this paper realized there's a hidden flaw. Just because an artist can describe a red car perfectly doesn't mean they can draw a red car when asked. Conversely, they might draw a beautiful red car but fail to recognize it as red when asked to describe it.

They call this a lack of Cross-Task Consistency. It's like a person who can recite a recipe perfectly but burns the cake when they try to cook it. The paper argues that current models often have this "split personality": their internal understanding of the world doesn't match their ability to create it.

The Solution: XTC-Bench (The "Fact-Check" System)
To fix this, the authors built a new testing ground called XTC-Bench. Instead of just asking general questions, they use a "Scene Graph."

Think of a Scene Graph as a detailed, structured recipe card for a picture. It breaks an image down into tiny, atomic facts:

Object: A car.
Attribute: It is white and silver.
Relation: It is parked in front of a tree.

Here is how their test works:

The Master Recipe: They start with a perfect "recipe card" (the Scene Graph) of a real photo.
The Artist's Turn: They feed the recipe to the AI and ask it to draw the picture.
The Guide's Turn: They show the AI the original photo and ask it to read back the recipe (e.g., "What color is the car?").
The Comparison: They compare what the AI drew against what the AI said.

If the AI draws a white car but says the car is blue, it has failed the consistency test, even if it drew a good car and gave a good answer separately.

The New Score: CCTA (The "Truth Meter")
They invented a new score called CCTA (Continuous Cross-Task Agreement).

Old Way: "Did you get the answer right?" (Accuracy).
New Way (CCTA): "Did you tell the truth in both directions?" (Consistency).

They also created a smarter score called AW-CCTA. This is crucial because it catches a specific trick: Consistent Hallucination.

Scenario: An AI is terrible at everything. It always draws a blue car, and when asked, it always says the car is blue.
Old Score: It looks consistent! (It matches itself).
New Score (AW-CCTA): It gives a low score because, while it is consistent, it is consistently wrong. It rewards models that are consistent and correct.

What They Found
The authors tested 8 open-source models and 2 commercial giants (like Google's Gemini and OpenAI's GPT). Here is what they discovered:

High Scores Don't Mean High Consistency: A model can be great at drawing and great at describing, but still have a "split personality." The two skills don't automatically talk to each other.
Understanding is Easier than Creating: Almost every model was better at describing a picture than drawing it. They are better guides than artists.
Architecture Matters, But Not How You Think: You might think that if you build a model with one brain for both tasks, it will be consistent. The authors found that simply sharing the "brain" (architecture) isn't enough. The key is how tightly the training goals for "seeing" and "drawing" are linked. If they aren't linked tightly, the model learns two different languages for the same world.
The "Consistent Wrong" Trap: Some models were found to be consistently hallucinating (making up the same wrong facts in both directions). The new AW-CCTA score was the only one that caught this.

The Takeaway
This paper introduces a new way to test AI that looks for internal honesty. It shows that just because an AI can do two things well doesn't mean it understands the world in a single, coherent way. To build truly unified AI, we need to stop testing the "Guide" and the "Artist" separately and start checking if they are telling the same story.

1. Problem Statement

Unified Multimodal Models (uMMs) aim to perform both visual understanding (perception) and visual generation (synthesis) within a shared architectural and representational space. While current benchmarks evaluate these capabilities independently (e.g., VQA for understanding, FID/CLIP scores for generation), they fail to assess semantic coherence between the two tasks.

The Core Issue: A model might excel at answering questions about an image (understanding) but fail to generate those same details when prompted (generation), or vice versa. This indicates a structural asymmetry in the model's internal representation. High scores on isolated tasks can mask a fundamental lack of unified semantic alignment, where the model learns to "hallucinate" consistently across tasks rather than grounding both in a shared reality.

2. Methodology: XTC-Bench

The authors introduce XTC-Bench, a scene-graph-grounded evaluation framework designed to measure cross-task visual semantic consistency.

A. Data Construction Pipeline

Source: 2,000 manually curated images from COCO 2017 and Visual Genome.
Scene Graph Extraction: A two-stage pipeline is used to create a "Reference Scene Graph" ( $G$ $G$ ) for each image:
1. Object Detection & Segmentation: Uses kMaX-DeepLab for panoptic segmentation.
2. Relation Prediction: Uses Fair-PSG to predict pairwise relations.
3. Refinement:
  - Filtering: High-confidence thresholds are applied to remove spurious relations.
  - VLM Verification: A large Vision-Language Model (Qwen3-VL-235B) verifies relations and attributes using bounding box annotations.
  - Attribute Generation: Uses a "Dual-View" approach (Semantic Spotlight + Detail Crop) to ensure accurate attribute recognition for specific objects.
Prompt & Query Generation:
- Generation Prompts: The scene graph is linearized into a natural language prompt for image generation.
- Understanding Queries: Fact-aligned VQA questions are generated for every atomic fact (object, attribute, relation) in the scene graph.

B. Evaluation Protocol

For a given model $M$ :

Generation: The model generates an image ( $\hat{I}$ ) from the prompt. This image is parsed back into a predicted scene graph ( $\hat{G}$ ) using the same extraction pipeline.
Understanding: The model answers the VQA questions based on the original image.
Matching: A two-stage semantic graph matching algorithm (using the Hungarian algorithm) aligns nodes and edges between the ground-truth graph $G$ and the predicted graph $\hat{G}$ .

C. Metrics

The framework introduces three key metrics:

Generation Score ( $G$ ) & Understanding Score ( $U$ ): Evaluated using an LLM-as-Judge (Qwen3-235B) that scores semantic equivalence (0–5 scale) between the model's output and the ground truth, rather than exact string matching.
Continuous Cross-Task Agreement (CCTA):
- Measures the consistency between generation and understanding responses for the same atomic facts.
- Formula: $CCTA = \frac{\sum (1 - |g_f - u_f|)}{\sum 1}$ .
- Limitation: A model that is consistently wrong (hallucinating the same error in both tasks) will score high on CCTA.
Accuracy-Weighted CCTA (AW-CCTA):
- Addresses the "consistent hallucination" flaw by scaling the consistency term by the mean joint accuracy.
- Formula: $AW\text{-}CCTA = \frac{\sum (1 - |g_f - u_f|) \cdot \frac{g_f + u_f}{2}}{\sum 1}$ .
- This metric penalizes models that are symmetrically incorrect, rewarding only those that are symmetrically correct.

3. Key Contributions

Formalization of Cross-Task Consistency: Defined consistency as a measurable property distinct from standalone accuracy, introducing CCTA and AW-CCTA to quantify representational symmetry.
XTC-Bench Framework: A fully automated, scene-graph-grounded benchmark that jointly evaluates generation, understanding, and their alignment under a shared structured representation.
Empirical Discovery: Demonstrated that current uMMs exhibit systematic structural asymmetry. High performance in one task does not guarantee alignment in the other, and architectural unification alone does not ensure semantic coherence.

4. Experimental Results

The authors evaluated 8 open-source models (e.g., BAGEL, OmniGen, Show-o, Janus-Pro) and 2 commercial models (Gemini, GPT-5).

Weak Correlation: There is a weak predictive relationship between standalone task performance ( $G$ or $U$ ) and cross-task consistency (AW-CCTA). A model can have high generation scores but low alignment.
The "Consistent Hallucination" Failure Mode:
- Models like MMaDA-8B showed a high raw CCTA (0.630) but a near-zero AW-CCTA (0.144). This indicates the model generates and understands the same incorrect facts, a failure mode invisible to standard metrics.
- BAGEL-7B and Gemini-2.5 Flash achieved the highest AW-CCTA, indicating better balanced performance.
Architectural Insights:
- Pure Autoregressive (AR) models (e.g., OmniGen, Janus-Pro) showed the highest mean AW-CCTA, suggesting that a unified next-token prediction objective naturally couples tasks better.
- AR + Diffusion models showed the largest gap between generation and understanding, indicating sensitivity to the calibration between the two components.
- Pure Diffusion models showed high raw consistency but low accuracy-weighted consistency.
Understanding > Generation: Most models scored higher on understanding than generation, suggesting current training objectives favor perceptual grounding over generative faithfulness.

5. Significance and Future Directions

Beyond Isolated Metrics: XTC-Bench proves that evaluating uMMs requires moving beyond isolated accuracy to measure internal representation coherence.
Diagnostic Tool: The divergence between CCTA and AW-CCTA serves as a critical diagnostic for "consistent hallucinations," a dangerous failure mode in safety-critical applications.
Guidance for Training: The results suggest that simply unifying architectures is insufficient; future research must focus on coupling learning objectives across modalities to ensure that perception and synthesis share the same semantic ground.
Limitations: The framework currently relies on automated teacher models (which may inherit biases) and static images. Future work should explore white-box probing to locate divergence points in the architecture and extend the framework to video and audio.

In conclusion, XTC-Bench provides a rigorous, reproducible method to expose the "semantic gap" in unified multimodal models, offering a concrete path toward developing models that truly understand and generate a consistent reality.