🤖 AI

GEBench: Benchmarking Image Generation Models as GUI Environments

GEBench is a new benchmark and a multi-dimensional metric (GE-Score) designed to evaluate the ability of image generation models to produce temporally coherent and logically consistent GUI state transitions across single-step and multi-step interactions.

Original authors: Haodong Li, Jingwei Wu, Quan Sun, Guopeng Li, Juanxi Tian, Huanyu Zhang, Yanlin Lai, Ruichuan An, Hongbo Peng, Yuhong Dai, Chenxi Li, Chunmei Qing, Jia Wang, Ziyang Meng, Zheng Ge, Xiangyu Zhang, Daxi

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haodong Li, Jingwei Wu, Quan Sun, Guopeng Li, Juanxi Tian, Huanyu Zhang, Yanlin Lai, Ruichuan An, Hongbo Peng, Yuhong Dai, Chenxi Li, Chunmei Qing, Jia Wang, Ziyang Meng, Zheng Ge, Xiangyu Zhang, Daxin Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a video game, but instead of a pre-made world, the game is being "dreamed up" in real-time by an AI. You press a button to "Open Settings," and the AI instantly paints a brand-new picture of what that settings menu should look like.

This sounds magical, but there is a huge problem: How do we know if the AI is actually "playing the game" correctly, or if it’s just hallucinating random, beautiful nonsense?

That is exactly what this paper, GEBench, is about. Here is the breakdown:

1. The Problem: The "Beautiful Liar" Paradox

Current AI models (like those that make art) are amazing at making pretty pictures. However, if you ask an AI to show you a "Phone Settings" screen and then ask it to "Click the Wi-Fi button," the AI might show you a beautiful picture of a phone, but the Wi-Fi button might be in the wrong place, the text might look like gibberish, or the next screen might look like a completely different phone.

In the AI world, we call this a lack of temporal coherence and spatial grounding. In everyday language, it means the AI has a "short-term memory problem" and "bad hand-eye coordination." It can draw a single frame perfectly, but it can't follow a logical story.

2. The Solution: GEBench (The Ultimate "UI Driving Test")

The researchers created GEBench, which acts like a rigorous driving test for AI. Instead of just asking the AI to "draw a cat," they give it specific "driving" tasks within a digital interface:

The Single Step: "Click this specific icon. Show me what happens next." (Testing basic reflexes).
The Long Journey: "Start at the home screen and successfully order a coffee." (Testing long-term planning and memory).
The Imaginary App: "Design a brand new app for a space traveler." (Testing creativity and logic).
The Precision Test: "Click exactly at these coordinates [X, Y]." (Testing hand-eye coordination).

3. The Scoring: The "GE-Score" (The Five-Star Review)

To grade the AI, they don't just say "looks good" or "looks bad." They use a five-dimensional grading system, much like a restaurant critic:

Goal Achievement (Did you get the food?): Did the AI actually do what you asked? If you asked to open a menu, is the menu open?
Interaction Logic (Does the kitchen make sense?): If you click "Delete," does the app actually show a "Are you sure?" pop-up, or does it just teleport to a random screen?
Consistency (Is the waiter the same person?): If you move from one screen to the next, does the phone still look like the same phone, or did the color and shape suddenly change?
UI Plausibility (Is the menu readable?): Do the buttons look like real buttons, or do they look like weird, melted blobs?
Visual Quality (Is the presentation pretty?): Is the text sharp and clear, or is it blurry and messy?

4. The Verdict: The AI is a "Talented but Distracted Student"

After testing the world's best AIs (like Google's and OpenAI's models), the researchers found something interesting:

The AIs are brilliant at single steps (they can draw a single, beautiful screen), but they fail miserably at long sequences. As the "story" goes on, the AI gets confused. It forgets where it was, it loses track of the coordinates, and it starts "hallucinating" text and icons.

Why does this matter?

In the future, we want AI "Agents" that can actually use your computer for you—booking flights, organizing files, or managing your emails. To do that, the AI needs to live in a reliable "world." GEBench is the yardstick that will help scientists build an AI that doesn't just dream of interfaces, but actually understands how to navigate them.

Technical Summary: GEBench: Benchmarking Image Generation Models as GUI Environments

1. Problem Statement

As image generation models advance, there is a growing potential to use them as Generative GUI Environments. Unlike traditional simulators that rely on fixed software stacks or hardware, generative models can simulate dynamic, diverse, and low-cost interaction trajectories by predicting future interface states based on user instructions.

However, existing benchmarks are ill-suited for this purpose:

General-domain benchmarks focus on visual fidelity (aesthetic realism) rather than functional logic.
Video generation benchmarks focus on continuous temporal transitions (natural movement), whereas GUIs involve discrete state jumps (action-triggered interface changes).
The Gap: There is a lack of evaluation frameworks that can assess the functional logic, state-transition consistency, spatial grounding, and structural integrity required for a model to act as a reliable GUI simulator.

2. Methodology

The authors propose GEBench, a comprehensive benchmark designed to evaluate how effectively image generation models can serve as interactive GUI environments.

A. Benchmark Design (Task Suites):
GEBench consists of 700 high-quality samples across five distinct task categories:

Single-step Visual Transition: Evaluates fine-grained instruction following for a single action.
Multi-step Planning: Assesses long-horizon planning and temporal coherence across a 5-step trajectory.
Zero-shot Virtual GUI (fiction-app): Tests out-of-distribution generalization by asking models to generate unseen layouts from scratch.
Rare Trajectory Synthesis (real-app): Evaluates logical reasoning in long-tail, data-scarce interaction scenarios.
Grounding-based Generation: Tests spatial awareness by requiring the model to predict a transition based on specific normalized coordinates $[0, 1000]$ .

B. Evaluation Metric (GE-Score):
To move beyond simple visual quality, the authors introduce GE-Score, a five-dimensional metric evaluated via Vision Language Models (VLMs) as judges (e.g., GPT-4o, Gemini). The dimensions are:

Goal Achievement (GOAL): Does the output satisfy the user's intent?
Interaction Logic (LOGIC): Is the transition plausible according to UI patterns?
Consistency (CONS): Are unaffected regions preserved (no unintended visual drift)?
UI Plausibility (UI): Are elements structurally coherent and native-looking?
Visual Quality (QUAL): Is the text readable and the rendering artifact-free?

C. Data Construction:
The pipeline involves capturing raw screen recordings, annotating actions into structured JSON metadata, and applying a three-stage quality control process (rule-based preprocessing, expert verification, and statistical calibration).

3. Key Contributions

GEBench Dataset: A specialized dataset of 700 samples covering diverse GUI interaction paradigms.
GE-Score Framework: A multi-dimensional, VLM-guided scoring rubric that prioritizes functional logic and temporal coherence over mere aesthetic beauty.
Systematic Analysis: A rigorous evaluation of 12 state-of-the-art models (8 commercial, 4 open-source) that identifies specific technical bottlenecks in generative GUI simulation.

4. Results and Findings

Performance Gap: While top commercial models (e.g., Nano Banana Pro and GPT-image-1.5) perform well on single-step transitions (scoring $>80$ ), their performance plummets in multi-step planning, often dropping below 60 or even 10 points.
The "Visual Over-optimism" Paradox: High visual fidelity (QUAL) does not guarantee functional viability. Models often generate beautiful images that contain hallucinated widgets or illogical layouts.
Critical Bottlenecks:
- Spatial Grounding: Models struggle to map abstract coordinates to precise pixels, leading to extremely low "Goal Achievement" scores in grounding tasks.
- Temporal Coherence: Errors accumulate during multi-step sequences, causing the trajectory to diverge from the intended goal.
- Semantic Rendering: Deficiencies in icon interpretation and text rendering (especially in complex or non-English layouts) lead to functional distortions.

5. Significance

GEBench shifts the evaluation paradigm for generative models from "looking real" to "acting logically." By identifying that current models fail at long-term planning and precise spatial grounding, the paper provides a roadmap for future research. The ultimate goal is to develop high-fidelity, temporally coherent generative GUI systems that can serve as scalable, synthetic training environments for autonomous GUI agents.