🤖 AI

Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

The paper introduces CEDI, a dynamic, multi-turn interaction framework that bridges the gap between static benchmarks and real-world applications by revealing significantly more and more realistic visual hallucinations in Multimodal Large Language Models compared to conventional evaluation methods.

Original authors: Yijiang Li, Huiqi Zou, Bingyang Wang, Ziang Xiao

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yijiang Li, Huiqi Zou, Bingyang Wang, Ziang Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to see the world. For a long time, scientists have tested these robots by showing them a single picture and asking, "What do you see?" It's like a pop quiz where the robot has to write a short essay about a photo. If the robot gets the main things right, it gets an A. But in the real world, we don't just stare at photos and write essays. We talk to our devices. We ask, "Is that a dog?" then, "What color is its collar?" then, "Wait, is that a dog or a fox?" We have conversations. We change our minds. We get confused. The problem is that our old "pop quiz" tests don't catch the mistakes robots make when they are actually chatting with us. They might pass the quiz but fail the conversation, making up facts that aren't there just to keep the chat going. This is a big deal because if a robot is driving a car or helping a doctor, making things up can be dangerous. We need a way to test them the way we actually use them: in a messy, back-and-forth chat.

This paper introduces a new way to test these "Vision Language Models" (VLMs) called CEDI. Think of CEDI not as a test, but as a three-person play. First, you have the Robot (the model being tested). Second, you have a Detective (an automated examiner). Third, you have a Judge (a grader). The Detective doesn't just show the Robot a picture and walk away. Instead, the Detective is given a secret map of the picture (called a "Scene Graph") and a specific goal, like "Find the parking spot." The Detective then starts a conversation with the Robot, asking questions that get trickier and more specific as the chat goes on. The Detective might ask, "Is there a red car?" and if the Robot says yes, the Detective might follow up, "What model is it?" or even try to trick the Robot by asking, "Why is the blue car parked there?" when there is no blue car. The goal is to see if the Robot will start making things up (hallucinating) to keep the conversation alive.

The authors found that this "Detective" method is much better at catching lies than the old "pop quiz" style. When they used CEDI on several different robots, they discovered that the robots made up significantly more fake details than the old tests showed. For example, on one dataset, the new method found that the robots' "hallucination rates" jumped by 5 to 23 percentage points compared to the old methods. The robots were also much more likely to lie when the conversation got long. It seems that as the chat goes on, the robots get confused by their own previous answers and start reinforcing their own mistakes, like a game of "telephone" where the message gets more and more wrong.

The paper also showed that the robots are terrible at saying "I don't know." When the Detective asked a question based on a fake premise—like "What is the man holding?" when there is no man in the picture—the robots often tried to answer anyway, inventing a man and a hat. They struggled most when they were supposed to reject a false idea. The new "Judge" in this system, which uses a special graph-matching score, was able to catch these lies more consistently than previous tools, matching human judges' opinions better than the old methods did.

In short, the paper suggests that the old way of testing robots with single questions is missing a huge chunk of their problems. By switching to a dynamic, multi-turn conversation that mimics real life, we can see that these robots are much more prone to making things up, especially when they are trying to keep a conversation going or when they are tricked by false assumptions. The authors aren't saying the robots are broken, but they are saying we need to stop testing them like they are taking a silent exam and start testing them like they are having a conversation, because that's where the real trouble hides.

Technical Summary: CEDI – Contextualized Evaluation of MLLMs through Dynamic, Multi-turn Interactions

Problem Statement

Despite substantial advances in Multi-modal Large Language Models (MLLMs) on static benchmarks, their effectiveness in real-world applications remains uncertain. The core issue is a fundamental misalignment between current evaluation paradigms and actual usage. Existing benchmarks are typically static, single-turn, and conducted in controlled settings, whereas real-world MLLM interactions are inherently dynamic, multi-turn, and contextualized. This gap prevents the detection of failure modes—specifically visual hallucinations (generating outputs unfaithful to image content)—that only emerge during complex, interactive dialogues. Furthermore, current evaluation methods often fail to capture the accumulation of errors over long contexts or the model's inability to reject false premises.

Methodology: The CEDI Framework

The authors propose CEDI (Contextualized Evaluations of MLLMs through Dynamic, multi-round Interactions), a framework that recasts evaluation as a three-party interaction:

The Evaluatee: The MLLM being tested.
The Examiner: An automated agent that conducts a semi-structured, multi-turn interview.
The Grader: An automated agent that scores the dialogue for faithfulness and coverage.

1. Dynamic and Contextualized Examiner

Inspired by human evaluative interviews, the examiner adapts its questioning strategy based on the model's responses.

Scene Graph Representation: To ensure questions are grounded in visual evidence rather than language priors, the examiner utilizes a Scene Graph ( $G = (V, E, A)$ ) representing objects, relations, and attributes.
Context Modeling: The examiner simulates a first-person user scenario with a specific goal (e.g., "I am visually impaired and need to find the restroom"). It selects a relevant sub-graph ( $G_s$ ) from the full scene graph to drive the conversation.
Traversal Strategy: The interview is modeled as a traversal over $G_s$ $G_{s}$ . At each turn, the examiner selects a target node and generates one of four question types:
- Regular: Grounded queries about observable elements.
- Follow-up: Questions conditioned on prior dialogue to test consistency.
- Adversarial: Questions about plausible but absent objects/relations to test reliance on visual evidence vs. statistical co-occurrence.
- Unanswerable: Questions based on fabricated premises (false objects/relations) to test the model's ability to reject false information.
Context Scaling: To ensure broad visual coverage, the examiner generates multiple distinct contexts per image, expanding the union of queried sub-graphs without requiring a single exhaustive caption.

2. Scene Graph-Based Grader

The grader evaluates the accumulated dialogue transcript rather than single-turn responses.

Parsing: An LLM parses the transcript into a predicted scene graph ( $\hat{G}$ ), filtering for physical entities.
Metrics: The grader compares $\hat{G}$ $\hat{G}$ against the ground-truth scene graph ( $G^*$ $G^{*}$ ) using:
- Graph Edit Distance (GED): Measures the cost of edit operations (insertions, deletions, substitutions) to transform $\hat{G}$ to $G^*$ . Insertions penalize hallucinations; deletions penalize missing content.
- $\Delta$ Con: Measures structural similarity via node-affinity matrices.
- Decomposition: The authors define $GED_{hal}$ (isolating hallucinated content) and $GED_{cov}$ (isolating missing content) to disentangle precision and recall.

Key Contributions

Framework Proposal: Introduction of CEDI, shifting MLLM evaluation from static benchmarks to dynamic, three-party interactions guided by scene graphs.
Questioning Strategy: Development of a semi-structured interview protocol with four distinct question types (including adversarial and unanswerable) designed to elicit specific hallucination behaviors.
Evaluation Metric: A scene-graph-based grader that quantifies both hallucination severity and visual coverage at the transcript level, offering higher alignment with human annotations than existing metrics like CHAIR or MMHal.
Empirical Validation: Comprehensive evaluation across diverse models (open-source and proprietary), datasets (VG, COCO, ADE20K), and domains.

Results

Increased Hallucination Elicitation: CEDI consistently exposes significantly more hallucinations than caption-based, binary-question (e.g., POPE), and prompt-only baselines. On the Synthetic Visual Genome (SVG) dataset, CEDI increased coverage by 3–19 percentage points and the per-turn hallucination rate (VALOR) by 5–23 percentage points compared to caption baselines.
Human Alignment: The $GED_{hal}$ metric shows a consistent Pearson correlation ( $\rho = 0.22–0.33$ ) with human-annotated hallucination counts across all models, outperforming CHAIR and MMHal, which fluctuate or show negative correlations for certain models.
Context and History Effects:
- Context Scaling: Increasing the number of contexts per image monotonically improves hallucination detection and coverage.
- Dialogue History: Models hallucinate more frequently when allowed to see their own conversation history, often reinforcing earlier unsupported claims.
- Question Types: Models are particularly vulnerable to "unanswerable" and "adversarial" questions, which require premise rejection.
Error Patterns: Qualitative analysis reveals that hallucinations are dominated by non-existent objects (62%), followed by wrong attributes (20%) and wrong relations (15%).

Significance and Claims

The paper claims that CEDI represents a step toward ecologically valid assessments of MLLMs. By simulating realistic, goal-driven, multi-turn interactions, CEDI reveals failure modes that static benchmarks miss, such as the accumulation of hallucinations over long contexts and the inability to refuse false premises. The authors argue that this framework provides a more faithful basis for diagnosing model failures and guiding the development of reliable VLMs for real-world deployment. While the current instantiation focuses on visual hallucination, the authors posit that the underlying formulation is generalizable to broader tasks and applications.