🤖 AI

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

The paper introduces MMBench-Live, a continuously evolving multimodal benchmark powered by a multi-agent automated pipeline that generates cost-effective, high-quality evaluation instances while preserving distribution consistency and mitigating data contamination.

Original authors: Yuanzhi Liu, Shousheng Zhao, Bo Zhou, Kongming Liang, Zhanyu Ma

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanzhi Liu, Shousheng Zhao, Bo Zhou, Kongming Liang, Zhanyu Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to grade a class of students who are learning to see and understand the world through computers (these are called Vision-Language Models, or VLMs).

For a long time, teachers have used the same old test papers (static benchmarks) to grade everyone. But there's a big problem: the students are studying the internet, and the internet is changing every second. If the test paper is from last year, the students might have already memorized the answers because those questions appeared in their study materials (data contamination). Also, the test might be outdated, not reflecting what the students can actually do today.

MMBench-Live is a new, revolutionary way of creating tests. Instead of printing a static paper and hoping it stays relevant, the authors built a robotic test-making factory that constantly churns out fresh, new questions.

Here is how it works, using simple analogies:

1. The "Recipe Book" (Structured Benchmark)

First, the team didn't just grab random pictures. They took the original test (MMBench) and turned it into a strict recipe book.

They broke down every question into its core ingredients: What skill is this testing? (e.g., reading a sign, counting objects, understanding a joke).
They identified the "flavor" of the original questions (e.g., "These questions usually involve busy city streets" or "These usually involve text on a menu").
This recipe ensures that even though the new questions are brand new, they taste exactly like the old ones. They test the same skills in the same way.

2. The "Smart Scout" (Task-Aware Data Acquisition)

Next, the system sends out a Smart Scout to find new pictures from the real world (the internet).

The Problem: If you just ask a search engine for "a picture of a cat," you might get a cartoon cat, a toy cat, or a cat sleeping. But if your test requires a "cat chasing a laser pointer," a generic search fails.
The Solution: The Scout has a specific mission. It goes out, finds pictures, and then has a Feedback Controller (a strict editor) check them.
The Loop: If the Scout brings back a picture of a sleeping cat, the editor says, "No, that doesn't fit the recipe. Try searching for 'active cats' instead." The Scout tries again. This happens automatically until the pictures match the "flavor" of the original test perfectly.

3. The "Chef and the Taste-Tester" (QA Generation & Verification)

Once the right pictures are found, the system needs to write the questions and answers.

The Chef: A team of AI "chefs" writes the questions and answers based on the picture.
The Taste-Tester: Here is the clever part. Usually, an AI might just guess the answer. But MMBench-Live forces the AI to write a step-by-step recipe (an executable plan) for how to get the answer.
The Verification: A separate, "blind" robot (which can't see the picture, only the text) follows that recipe. It runs the steps. If the recipe says "Count the red cars" and the robot counts them and gets 3, but the answer key says 4, the system knows the answer is wrong and throws it out. This ensures the answers are mathematically and logically correct, not just lucky guesses.

4. The Results: A Fast, Cheap, and Fair Test

The paper claims this system is a game-changer for three reasons:

It's Fresh: It creates 5,900 new test questions using real-world data from the internet.
It's Cheap and Fast: Doing this manually would cost thousands of dollars and take months. This robot factory does it in 1–2 hours for about $30.
It's Fair: Because the questions are new and generated on the fly, the students (AI models) can't just memorize the answers from their training data. The paper found that the "memorization signals" (cheating via memory) were much weaker here than on old tests.

The Bottom Line

Think of MMBench-Live as a live cooking competition where the ingredients are delivered fresh every hour, and the judges have a strict checklist to ensure the dish matches the original recipe. It proves that we can keep testing AI models fairly and accurately without getting stuck with outdated, memorized, or expensive tests. It's a sustainable way to see if AI is actually getting smarter or just better at memorizing the past.

Technical Summary: MMBench-Live

Problem Statement

Current vision-language model (VLM) evaluation relies heavily on static benchmarks. While essential for assessment, these fixed test sets suffer from temporal staleness, data contamination (where training corpora overlap with test data, inflating scores), and prohibitive maintenance costs. As VLMs scale and are trained on ever-growing web corpora, static benchmarks become increasingly fragile proxies for true model capability.

Existing attempts to update benchmarks face significant trade-offs:

Self-evolving methods (visual perturbations/linguistic rewrites) often suffer from semantic drift and reduced comparability.
Generation-based methods can introduce distributional gaps under complex constraints.
Real-world data updating is promising for freshness but often lacks the systematic structure to preserve task semantics, distributional alignment, and cross-version comparability across diverse multimodal tasks.

The core challenge is to create a benchmark that is continuously evolving with temporally fresh data while maintaining semantic coherence, distributional consistency, and evaluation stability comparable to the original static version.

Methodology: MMBench-Live Framework

The authors propose MMBench-Live, a continuously evolving multimodal benchmark constructed via a multi-agent-driven automated pipeline. The framework treats benchmark evolution as a task-guided dataset construction process, instantiated by updating the original MMBench.

The pipeline consists of three primary stages:

1. Structured Benchmark Specification

To ensure consistency, the original MMBench is converted into a structured description containing four components:

Evaluation Purpose: High-level semantic characterization (e.g., spatial understanding, compositional inference).
Evaluation Format: Structural definitions of input/output modalities and response types.
Task Hierarchy: A multi-level decomposition from coarse-grained tasks to fine-grained sub-tasks.
Atomic Tasks: Leaf-level units defining the smallest operational evaluation units. Crucially, each atomic task includes a Task Characterization with:
- Textual: QA examples clarifying semantics.
- Visual: Task-related visual patterns (recurrent visual/semantic attributes not explicitly defined but implicitly constraining valid instances). These are identified via a data-driven procedure using a VLM to extract patterns and an LLM to consolidate them.

2. Task-Aware Data Acquisition

Instead of generic search, data collection is constrained by task intent and visual evidence. This stage employs a Feedback-Controlled loop:

Acquisition Planner: Generates an executable plan ( $P_t$ ) including observable fields, acquisition strategy, and query sets.
Acquisition Executor: Supports three retrieval modes:
1. Source-specific Structured Retrieval: Crawling designated authoritative sites.
2. Keyword-Guided Semantic Retrieval: Using Google Image API with a 1-year temporal window to ensure freshness.
3. Open-Domain Retrieval: Sampling from Flickr API for broad, natural coverage.
Feedback Controller: A lightweight multi-modal judge ( $J$ $J$ ) evaluates retrieved candidates against the task purpose ( $G_t$ $G_{t}$ ) and visual patterns ( $B_t$ $B_{t}$ ).
- If the acceptance rate ( $\rho_q$ ) of top- $K$ candidates is below a threshold ( $\tau$ ), the controller aggregates visual rationales from rejected samples to refine the search query ( $q_{m+1}$ ).
- This loop runs for a maximum of three iterations to correct poorly aligned queries before final collection.

3. QA Generation and Verification

To ensure reliability, the system generates Question-Answer (QA) pairs with executable reasoning:

Generation: Multiple multimodal models generate candidate QA triplets $(q, a, \pi)$ , where $\pi$ is an executable solution plan. The best candidate is selected via model comparison and rule-based checks.
Executable Reasoning Verification: To mitigate visual hallucinations, verification is decoupled from direct vision-based judgment.
- The solution plan $\pi$ is treated as a sequence of executable operations.
- Tools (e.g., object detection, depth estimation, OCR) execute these operations on the image to produce textual intermediate outputs.
- A vision-blind language model verifier predicts the answer based only on the question and the tool-derived text.
- The generated answer is accepted only if it matches the verified outcome, ensuring the QA pair admits a deterministic validation procedure.

Key Contributions

MMBench-Live Benchmark: A continuously evolving benchmark constructed via a multi-agent automated pipeline, enabling scalable, low-cost updates with temporally fresh real-world data.
Distribution-Consistent Update Strategy: A novel approach combining task-related visual pattern identification with feedback-controlled data acquisition. This preserves task semantics and visual characteristics across benchmark versions, addressing the "semantic drift" problem in dynamic updates.
Systematic Evaluation: Comprehensive analysis demonstrating high QA correctness, low construction costs, stable cross-version consistency, and reduced contamination-related memorization signals compared to static benchmarks.

Experimental Results

The framework was instantiated on MMBench, generating 5.9K new evaluation instances.

QA Quality: Manual verification of 10% of samples across tasks yielded a 96.06% answer correctness rate.
Cost and Efficiency: Each update cycle (1–2 hours) costs approximately $30, a reduction of multiple orders of magnitude compared to manual pipelines (which typically cost thousands of dollars and take months).
Model Performance: Evaluations on representative open-source VLMs (e.g., Qwen2.5-VL, DeepSeek-VL) showed that MMBench-Live preserves stable model rankings. While specific models excel in different dimensions (e.g., DeepSeek-VL in coarse perception, Qwen2.5-VL in fine-grained reasoning), the overall relative ordering remains consistent.
Cross-Version Consistency:
- Task-Model Level: Pearson correlation of 0.52 and Spearman correlation of 0.43 indicate that models performing well on the original benchmark tend to maintain performance on the updated version.
- Model-Averaged Level: Stronger correlations (Pearson 0.89, Spearman 0.83) confirm that global model rankings are preserved.
Contamination Analysis: Using the PaCoST framework to measure memorization signals (confidence shifts between original and paraphrased questions), MMBench-Live exhibited weaker memorization-related confidence bias than the original MMBench, suggesting reduced susceptibility to data contamination.

Significance and Claims

The paper claims that MMBench-Live provides a practical, scalable, and low-cost paradigm for sustainable multimodal benchmark evolution. By treating benchmark updates as a structured, task-guided construction process rather than simple sample replacement, the framework successfully balances the need for temporal freshness with the necessity of evaluation stability.

The authors emphasize that their approach mitigates the "fragility" of static benchmarks in the era of rapidly scaling VLMs. The introduction of executable reasoning verification and feedback-controlled acquisition ensures that the benchmark remains a reliable proxy for model capability without incurring the prohibitive costs of manual curation. The results suggest that dynamic benchmarks can surface fine-grained capability differences while maintaining the integrity of cross-version comparisons, offering a viable path forward for the community to combat data contamination and temporal staleness.