💻 computer science

CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions

This paper introduces CREval, an automated and interpretable QA-based evaluation pipeline, and CREval-Bench, a comprehensive benchmark with over 800 samples, to systematically assess and reveal the current limitations of state-of-the-art image editing models in handling complex and creative manipulation tasks.

Original authors: Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chonghuinan Wang, Zihan Chen, Yuxiang Wei, Tianyi Jiang, Xiaohe Wu, Fan Li, Wangmeng Zuo, Hongxun Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical art studio where you can tell a computer, "Turn this photo of my dog into a superhero wearing a cape made of clouds," and it tries to do exactly that. This is what Instruction-Based Image Editing is all about.

But here's the problem: How do you know if the computer actually did a good job? Did it listen to your instructions? Did it keep your dog looking like your dog? Is the picture blurry or weird?

Currently, checking these results is like trying to grade a student's essay by just looking at the final score without reading the teacher's notes. It's vague, and sometimes the "teacher" (the AI evaluator) is biased or doesn't explain why it gave a bad grade.

This paper introduces CREval, a new, super-smart way to grade these AI art projects. Think of it as hiring a detective instead of just a judge.

The Problem: The "Black Box" Judges

Before this paper, researchers used big AI models to look at an edited image and just say, "Score: 7/10."

The Issue: We didn't know why it was a 7. Did the AI miss the cape? Did the dog look like a cat? The score was a "black box"—a mystery.
The Gap: Existing tests were mostly for simple tasks like "remove the person" or "change the sky to blue." They weren't built for creative, wild ideas like "turn the warrior into a tropical fish plushie" or "make a comic strip about a cowboy horse."

The Solution: CREval (The Detective)

The authors created CREval, which stands for Creative Review Evaluation. Instead of asking the AI, "How good is this?", CREval asks the AI a series of specific Yes/No questions (like a quiz).

Imagine you are a teacher grading a student's drawing. Instead of just giving a grade, you ask:

"Did the student draw the hat?" (Yes/No)
"Is the hat blue, as requested?" (Yes/No)
"Did they keep the student's original face?" (Yes/No)
"Are the lines shaky and messy?" (Yes/No)

CREval does exactly this, but for AI images. It breaks the evaluation down into three main categories:

Instruction Following (IF): Did the AI actually do what you asked? (e.g., "Did it turn the robot into a cake?")
Visual Consistency (VC): Did the AI keep the important parts of the original photo? (e.g., "Is it still the same robot, just a cake robot, or did it turn into a totally different character?")
Visual Quality (VQ): Does the picture look good? (e.g., "Are the hands weird? Is the texture blurry?")

The Playground: CREval-Bench

To test this new detective system, the authors built a giant playground called CREval-Bench.

It contains over 800 complex, creative challenges.
These aren't simple tasks. They are things like: "Turn this wedding photo into a traditional Chinese cartoon," or "Make this mountain look like a giant stone titan rising from the ground."
It's like a "final exam" for AI art models, designed to be much harder than previous tests.

What Did They Find? (The Report Card)

They ran the top AI models (both free/open-source and paid/closed-source) through this new test. Here's the verdict:

The Good News: The paid, "closed-source" models (like Seedream 4.0 and Gemini 2.5) are currently the best at following these wild instructions. They are like the top art students who can handle complex prompts.
The Bad News: Even the best models struggle. They often fail to keep the original character's identity (Visual Consistency). For example, they might turn a person into a cake, but the cake looks nothing like the person.
The Open-Source Hope: Free models are catching up fast! Models like Qwen-Image-Edit are doing surprisingly well, showing that the gap between free and paid tools is shrinking.

Why Does This Matter?

Think of CREval as a universal translator between human creativity and AI capability.

For Developers: It tells them exactly where their AI is failing (e.g., "You're great at following instructions, but you keep messing up the faces").
For Users: It gives us a reliable way to know which AI tool is actually good at handling our creative, weird, and wonderful ideas.

In short, CREval stops us from guessing if AI art is good. It gives us a clear, step-by-step report card that explains exactly what the AI did right and what it got wrong, helping us build better tools for creative expression.

1. Problem Statement

While instruction-based multimodal image manipulation has advanced rapidly, current evaluation methods fail to adequately assess complex and creative editing tasks. Existing benchmarks primarily focus on simple operations (e.g., object addition, removal, color adjustment) or rely on "black-box" scoring by Multimodal Large Language Models (MLLMs). These approaches suffer from:

Lack of Interpretability: Direct MLLM scoring often lacks transparent rationales, making it unclear why a score was assigned.
Incomplete Coverage: Holistic scoring may miss specific dimensions of failure, such as subtle visual artifacts or partial instruction adherence.
Inability to Handle Creativity: Current benchmarks do not sufficiently test "free-style" creative tasks (e.g., transforming a warrior into a plush toy, creating surreal fantasy scenes) which require deep semantic understanding and structural reconfiguration.

2. Methodology

The authors propose CREval, a fully automated, interpretable evaluation pipeline, and CREval-Bench, a comprehensive benchmark.

A. CREval-Bench (The Benchmark)

Scale: Contains over 800 editing samples and 13,000 evaluation queries.
Taxonomy: The benchmark categorizes creative editing into 3 main levels and 9 specific dimensions:
1. Customization: Derivative Characters (e.g., chibi), Reimagined Representations (e.g., stamps/postcards), Surreal Fantasy.
2. Contextualization: Containerized Scenarios (e.g., snow globes), Commercial Design (e.g., packaging), Informational & Narrative Expression (e.g., comics).
3. Stylization: Artistic Style Transfer, Identity & Cultural Transformation, Material Transformation (e.g., turning objects into LEGO or ceramic).
Construction: High-quality source images are curated, and creative instructions are generated using GPT-4o with few-shot learning to ensure alignment with the 9 dimensions.

B. CREval (The Evaluation Framework)

Instead of asking an MLLM to assign a single holistic score, CREval decomposes evaluation into a Question-Answer (QA) based pipeline using three core metrics:

Instruction Following (IF): Verifies if the output accurately reflects the specific modifications requested.
Visual Consistency (VC): Measures the preservation of critical visual elements from the source image (e.g., identity, key features). Elements are assigned importance weights ( $w \in \{1, 2, 3\}$ ) based on their contribution to identity recognition.
Visual Quality (VQ): Assesses realism, structural coherence, and the absence of artifacts (e.g., distorted limbs, unnatural textures).

The Process:

QA Generation: For each image-instruction pair, an MLLM (using Chain-of-Thought reasoning) generates specific Yes/No questions and reference answers for IF, VC, and VQ.
Scoring: A target MLLM (e.g., GPT-4o) evaluates the edited image against these questions. If the model's answer matches the reference, points are awarded.
Final Score Calculation: The final score is a weighted average:
$S = 0.4 \times S_{IF} + 0.4 \times S_{VC} + 0.2 \times S_{VQ}$
Note: VQ is weighted lower (0.2) because current MLLMs are less sensitive to subtle visual artifacts compared to instruction adherence and consistency.

3. Key Contributions

CREval Framework: A novel, fully automated, QA-based evaluation pipeline that overcomes the opacity of direct MLLM scoring, providing fine-grained, interpretable metrics.
CREval-Bench: The first comprehensive benchmark specifically designed for creative image manipulation under complex instructions, covering 9 diverse dimensions.
Systematic Evaluation: Extensive experiments on 13 state-of-the-art models (both open-source and closed-source), revealing specific strengths and weaknesses in handling creative tasks.
Human Alignment: Validation through user studies demonstrating strong consistency between CREval's automated scores and human preference judgments.

4. Experimental Results

The paper evaluates 13 models, including Seedream 4.0, Gemini 2.5 Flash Image, GPT-Image-1, Qwen-Image-Edit, and FLUX.1 Kontext.

Overall Performance: Closed-source models generally outperform open-source ones, but the gap is narrowing.
- Top Performer: Seedream 4.0 achieved the highest overall score, showing a strong balance across IF, VC, and VQ.
- Runner-up: Qwen-Image-Edit (open-source) ranked second, outperforming GPT-Image-1 in aggregate performance.
- GPT-Image-1: While strong in Instruction Following (IF), it struggled significantly with Visual Consistency (VC), often losing key visual elements of the source image.
Key Challenges Identified:
- Visual Consistency: All models struggle to preserve key identity features when performing complex transformations (e.g., changing material or style).
- Complex Instructions: Models often fail to execute multi-step creative instructions (e.g., "Surreal Fantasy" or "Narrative Expression") without omitting details or hallucinating.
- Open-Source Progress: Models like Qwen-Image-Edit and FLUX.1 Kontext show promising progress, suggesting open-source models may soon rival closed-source capabilities.

5. Significance

Reliable Benchmarking: CREval provides a standardized, objective foundation for evaluating image editing models on creative tasks, moving beyond simple object manipulation.
Interpretability: By breaking down scores into IF, VC, and VQ with specific QA rationales, researchers can pinpoint exactly where a model fails (e.g., "it followed the style but lost the identity").
Future Research Direction: The results highlight that while instruction adherence is improving, visual consistency remains the primary bottleneck for creative image editing. This directs future research toward better identity preservation and structural coherence in generative models.
Community Impact: The release of CREval-Bench and the evaluation code enables the community to systematically track progress in this rapidly evolving field.