🤖 AI

Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity

This paper introduces MM-Eval, a unified framework that holistically assesses multimodal summaries by integrating factual text quality, image-text alignment, and visual diversity into a single interpretable metric calibrated to human preferences, thereby addressing the limitations of fragmented unimodal evaluation methods.

Original authors: Abid Ali, Diego Molla-Aliod, Usman Naseem

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abid Ali, Diego Molla-Aliod, Usman Naseem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a food critic reviewing a new restaurant. In the past, critics might have only tasted the soup (the text) or only looked at the garnish (the images) separately. They would give the soup a score and the garnish a score, but they wouldn't tell you if the garnish actually matched the soup or if the soup was lying about its ingredients.

This paper, "Measuring What Matters Beyond Text," argues that this old way of judging is broken. The authors, from Macquarie University, introduce a new system called MM-Eval to judge "Multimodal Summaries"—which are news stories or articles that come with both text and pictures.

Here is how their new system works, using simple analogies:

The Problem: The "Silo Effect"

Currently, computers judge these summaries in "silos" (separate rooms):

Text Room: They check if the words match a reference story. But this is like checking if two sentences use the same words, even if one says "The protest happened on May 3rd" and the other says "The protest happened on May 5th." The computer sees they are 90% similar and gives a high score, even though the date is wrong.
Image Room: They check if the computer picked the exact same photo a human picked. If a human picked a photo of a crowd from the left, and the computer picked a photo of the same crowd from the right, the computer gets a zero score, even though the meaning is perfect.
The Missing Link: The old systems don't check if the picture actually matches the story. You could have a perfect story about a flood and a perfect picture of a sunny beach, and the old system would give you a high score for both separately, missing the fact that they don't go together.

The Solution: MM-Eval (The "Three-Legged Stool")

The authors built MM-Eval to look at the summary as a whole, using three specific "legs" or pillars. If one leg is weak, the stool wobbles.

1. The Text Leg (Quality & Truth)

This leg checks if the story is good and, more importantly, true.

The "Fact-Checker" Robot: Instead of just counting matching words, the system breaks the summary down into tiny, atomic facts (like "The event was on Tuesday"). It then checks each tiny fact against the original source document. If the source says Tuesday and the summary says Wednesday, the system catches the lie.
The "Flow" Robot: It also checks if the story reads smoothly and makes sense, like a human editor would.

2. The Alignment Leg (Do the pictures fit the story?)

This leg asks: "Does this picture actually help explain the text?"

The "Judge" Robot: The system uses a smart AI (a Multimodal Large Language Model) to look at the text and the picture together. It acts like a judge in a courtroom, reasoning: "The text says 'flooding,' and the picture shows water. Good match." Or, "The text says 'flooding,' but the picture shows a parade. Bad match."

3. The Diversity Leg (Are the pictures unique?)

This leg checks if the pictures are just copies of each other.

The "Variety" Meter: Imagine a news story about a protest. If the computer picks three photos that are all taken from the exact same angle, just one second apart, it's boring and adds no new information. MM-Eval uses a mathematical trick (called "Entropy") to measure how different the pictures are from each other in their "meaning." If they are too similar, the score goes down.

The Big Discovery: The "Gatekeeper" Effect

After testing their system on thousands of news summaries, the authors found something surprising about how humans judge quality. They call this the "Text-Dominant Hierarchy."

Think of Factual Consistency (truthfulness) as a Gatekeeper.

If the text contains a lie (a hallucination), the Gatekeeper slams the door shut. No matter how beautiful the pictures are, or how diverse they are, the summary gets a terrible score.
Only if the text is truthful does the Gatekeeper open the door, allowing the pictures to add value.

In this specific world of news summaries, truth is the most important thing. Once the truth is established, the pictures matter, but they are secondary. The system learned that humans care about facts first, then the flow of the story, and finally, how well the pictures fit.

How It All Comes Together

The authors didn't just guess these weights; they trained a "learning model" on human ratings. They taught the computer to mimic how humans rank summaries.

They found that Text Quality accounts for about 79% of the final score.
Image Relevance (do the pics fit?) accounts for about 15%.
Visual Diversity (are the pics different?) accounts for about 6%.

Why This Matters

This new tool, MM-Eval, is like a smart, fair judge that doesn't get fooled by surface-level tricks. It doesn't need a perfect "answer key" (reference summary) to work; it can judge a summary based on the original source and the output alone.

The authors conclude that if you are building an AI to write news summaries, you should focus 100% on making sure the facts are right first. Once the facts are solid, then you can worry about picking the best pictures. If you try to make the pictures perfect while the text is lying, the whole summary fails.

In short: MM-Eval is a new way to grade AI summaries that says, "Don't just count the words or the pixels. Check if the story is true, if the pictures match the story, and if the pictures aren't all the same."

Technical Summary: MM-Eval – A Unified Framework for Multimodal Summarization Evaluation

1. Problem Statement

Multimodal Summarization with Multimodal Output (MSMO) aims to generate concise textual summaries paired with salient visuals from multimodal sources. While Multimodal Large Language Models (MLLMs) have advanced the capability to generate such outputs, the evaluation of MSMO remains fragmented. Current protocols suffer from a "Silo Effect," where text quality, image–text alignment, and visual diversity are assessed in isolation using unimodal metrics.

Key limitations of existing approaches include:

Text Metrics: Traditional metrics like ROUGE rely on lexical overlap, failing to capture semantic equivalence or factual consistency. Summaries with hallucinated facts can still achieve high scores if they superficially match reference texts.
Image Metrics: Image Precision (IP) penalizes semantically valid alternative images that do not exactly match human-annotated references and fails to account for alignment with the generated text.
Cross-Modal Gap: No existing metric effectively assesses the interaction between modalities. A system may score well on text and image selection independently while producing disconnected, misaligned multimodal outputs. Furthermore, current metrics do not penalize visual redundancy (e.g., selecting near-duplicate images).

2. Methodology: The MM-Eval Framework

To address these gaps, the authors introduce MM-Eval, a unified, modular evaluation framework that integrates three core dimensions: Text Quality, Cross-Modal Alignment, and Visual Diversity. The framework is designed to be reference-weak (relying minimally on reference summaries) and domain-portable.

2.1 Pillar 1: Text Quality ( $S_{text}$ )

This pillar assesses the generated text ( $T_{gen}$ ) against the source document ( $T_{source}$ ) using a "decompose-then-verify" approach:

Factual Consistency ( $S_{fact}$ ): Utilizing OpenFActScore, the system decomposes the summary into atomic facts and validates each against the source using an LLM. The score is the average precision of supported facts, robust to paraphrasing.
Qualitative Attributes: Using G-Eval, the system assesses coherence, fluency, and relevance via a structured Chain-of-Thought (CoT) prompting protocol to reduce variance.
Aggregation: $S_{text}$ is a weighted sum of these four components ( $S_{fact}$ , relevance, coherence, fluency).

2.2 Pillar 2: Cross-Modal Alignment ( $S_{relevance}$ )

This pillar measures whether the selected images ( $V_{sel}$ ) semantically support the generated text ( $T_{gen}$ ).

MLLM-as-a-Judge: An MLLM is instructed to perform CoT reasoning to evaluate the pragmatic relevance and complementarity of the image set relative to the text. This approach captures the integrated quality of the summary better than simple cosine similarity.

2.3 Pillar 3: Visual Diversity ( $S_{diversity}$ )

This pillar penalizes visual redundancy to ensure the image set provides distinct informational value.

Truncated CLIP Entropy (TCE): The system generates CLIP embeddings for the selected images, computes their empirical covariance matrix, and calculates the Von Neumann entropy of the normalized eigenvalues. This measures the "semantic volume" spanned by the images in latent space, penalizing sets where images convey highly overlapping semantic content.

2.4 Learned Aggregation

The final score is not a simple average but a learned linear combination of the three pillars:
$\text{MM-Eval} = \beta_0 + \beta_1 S_{text} + \beta_2 S_{relevance} + \beta_3 S_{diversity}$
The weights ( $\beta$ ) are learned via Ridge Regression (L2 regularization) on the mLLM-EVAL news benchmark, which contains $\sim$ 1,500 expert annotations of "Overall Quality." This allows the framework to align component contributions with human preferences specific to the domain.

3. Key Results and Analysis

The authors analyzed the mLLM-EVAL dataset and the performance of MM-Eval to derive several insights:

3.1 Text-Dominant Hierarchy

Analysis of the learned weights reveals a text-dominant preference structure in the news domain:

Text Quality ( $S_{text}$ ) accounts for approximately 79% of the total aggregate weight.
Within text quality, Factual Consistency is the primary driver, contributing roughly 43.5% of the total score.
Image-to-Text Relevance and Visual Diversity contribute smaller, complementary shares (approx. 15% and 6%, respectively).

3.2 The "Gatekeeper" Effect of Factual Consistency

Statistical analysis of human annotations reveals a bimodal distribution for factual consistency. The data suggests that factual consistency acts as a "gatekeeper":

If a summary contains significant hallucinations (low consistency), the overall quality score drops sharply, regardless of fluency or image quality.
Conversely, high factual integrity is a prerequisite for high overall ratings.
The learned weights reflect this non-linear human response, sharply penalizing factual failures rather than smoothing them via uniform averaging.

3.3 Conditional Value of Visuals

While visual components show weak or even negative marginal correlations with human rankings in isolation, ablation studies demonstrate that removing them significantly degrades the model's agreement with human rankings (e.g., removing $S_{relevance}$ drops Kendall's $\tau$ by 54%).

This indicates that visual relevance and diversity provide conditional signals: they refine quality distinctions when textual fidelity is already adequate but may co-vary with lower-quality outputs when considered in isolation.

3.4 Human Validation

A supplementary human evaluation on 200 samples confirmed that annotators value image relevance and diversity highly (mean scores comparable to text). This validates that the text-dominant weighting in MM-Eval reflects the marginal contribution of each component in news summarization (where text carries the primary informational load), rather than human indifference to visual quality.

4. Significance and Contributions

The paper claims the following significance for the field:

Unified Evaluation: MM-Eval is the first framework to jointly assess text quality, cross-modal alignment, and visual diversity in a single, interpretable metric, moving beyond the "Silo Effect" of unimodal evaluation.
Reference-Weak and Modular: Unlike metrics requiring reference summaries (e.g., ROUGE, IP), MM-Eval's components operate on the source document and system output alone. Its modular design allows individual scorers (e.g., the MLLM judge or TCE) to be upgraded or replaced as better models become available without altering the framework.
Data-Driven Weighting: By learning aggregation weights from human preferences, MM-Eval captures the implicit trade-offs humans make (e.g., prioritizing facts over fluency) rather than assuming uniform importance across dimensions.
Practical Guidance for System Design: The findings suggest a clear optimization hierarchy for MSMO systems: prioritize factual grounding first, followed by textual structure, and finally refine visual alignment and diversity. This is particularly relevant for text-dominant domains like news.
Interpretability: The framework provides an interpretable breakdown of scores, allowing developers to diagnose specific failures (e.g., poor diversity vs. hallucination) rather than relying on opaque "black box" model judgments.

5. Limitations and Scope

The authors maintain a modest stance regarding the framework's scope:

Domain Specificity: The learned weight hierarchy is shaped by the news domain. In image-dominant domains (e.g., product reviews), the prioritization of modalities may differ, requiring recalibration of the aggregation weights.
Proxy Noise: The automatic proxies for visual relevance and diversity (MLLM judge, TCE) may not fully capture human perception, as evidenced by weaker automatic correlations compared to human agreement.
Comparative Utility: With a moderate rank correlation ( $\tau \approx 0.37$ ) against human judgments, MM-Eval is best suited for coarse-to-medium system comparisons. For extremely close system comparisons, targeted human evaluation remains necessary.
Computational Cost: The reliance on multiple LLMs and MLLMs increases computational cost compared to lexical metrics, though this is deemed acceptable for offline evaluation.