🤖 machine learning

Generative Visual Code Mobile World Models

The paper introduces gWorld, a novel paradigm for mobile GUI world models that leverages a single Vision-Language Model to predict the next interface state as executable web code rather than raw pixels, thereby achieving high-fidelity visual rendering and precise text generation while significantly outperforming much larger existing models in accuracy.

Original authors: Woosung Koh, Sungjun Han, Segyu Lee, Se-Young Yun, Jamin Shin

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Woosung Koh, Sungjun Han, Segyu Lee, Se-Young Yun, Jamin Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blurry Crystal Ball"

Imagine you are trying to teach a robot how to use your smartphone. To do this, the robot needs a "crystal ball" (a World Model) that can predict: "If I tap this button, what will the screen look like next?"

Current attempts at building this crystal ball have a major flaw:

The "Text-Only" Model: Some models try to describe the next screen using words (e.g., "The button turns blue"). This is fast, but it's like describing a movie by reading the script. You lose all the visual details, like the exact font, the color shade, or the layout.
The "Pixel-Generator" Model: Other models try to draw the next screen pixel-by-pixel, like an artist painting a picture. While this looks pretty, these models are terrible at drawing text. They often produce gibberish letters, distorted buttons, or weird layouts. To fix this, they need a huge, slow, multi-step factory involving other AI tools to check the text and fix the errors. It's like hiring a painter, then a spell-checker, then an architect, then a editor just to draw a single button.

The Solution: The "Architect's Blueprint"

The authors propose a new way to build the crystal ball. Instead of asking the AI to draw the picture (pixels) or describe it (text), they ask it to write the code that builds the picture.

Think of it like this:

Old Way (Pixel Generator): Asking an artist to paint a perfect replica of a website. They might get the colors right, but the text might look like scribbles.
New Way (gWorld): Asking a master architect to write the blueprint (HTML/CSS code) for the website.

Because the AI is writing code (which is structured and logical), it knows exactly how to place a button, how to spell a word, and how to arrange the layout. When this code is "rendered" (run in a browser), it instantly becomes a perfect, crisp image of the next screen.

What is gWorld?

gWorld is the name of the new AI system the authors built.

It's a "World Model": It takes a current screen and an action (like "tap here") and predicts the next screen.
It speaks "Code": Instead of outputting an image file, it outputs web code (HTML/CSS).
It's Open and Fast: The authors released the "weights" (the brain) of the model for free. Because it skips the slow, complex factory of other methods, it is incredibly fast—predicting the next screen in less than a second.

How Did They Teach It?

You can't just ask an AI to write code for a screen it hasn't seen before. The authors had to build a special training factory:

Recycling Old Data: They took existing records of humans using phones (trajectories of taps and swipes).
The Translator: They used a super-smart AI to look at the "after" picture of a human action and translate that picture into clean, working code.
The Reasoning Step: They also taught the AI to "think out loud" first. Before writing the code, it explains why the screen will change (e.g., "The user clicked 'Send', so the email should disappear and a 'Sent' message should appear").

The Results: Why It Matters

The authors tested gWorld against 8 other top-tier AI models (some of which are 50 times larger and more expensive).

Accuracy: gWorld was the most accurate at predicting the next screen. It got the text right and the layout perfect.
Efficiency: It achieved these results with a much smaller model size. It's like a compact sports car beating a massive truck in a race.
The "Pareto Frontier": In the charts, gWorld created a new "best possible" line. You can't get better accuracy without making the model much bigger, and gWorld is already the best at its size.
Real-World Test: They even tested it on apps and languages (like Korean) it had never seen before, and it still performed very well.

The "Magic" of the Blueprint

The paper highlights a key insight: Mobile screens are mostly structure and text, not complex photos.
While some screens have photos (like a camera view), most are lists, buttons, and text. Because gWorld writes code, it treats the screen like a structured document. This means it never makes spelling mistakes or layout errors, which is the biggest weakness of the "pixel-painting" models.

Summary

The paper introduces gWorld, a new type of AI that predicts how a phone screen will change after a user interacts with it. Instead of trying to "paint" the next screen (which leads to blurry text and errors), gWorld writes the code to build the screen. This approach is faster, more accurate, requires less computing power, and produces perfect text and layouts, setting a new standard for how AI agents can learn to use our phones.

Technical Summary: Generative Visual Code Mobile World Models

Problem Statement

Mobile Graphical User Interface (GUI) World Models (WMs) aim to predict future interface states to enhance agent performance during training and inference. Current approaches face a critical trade-off:

Text-based WMs: Sacrifice visual fidelity by abstracting the pixel space into text, losing critical spatial layout and visual attributes (e.g., iconography, typography).
Visual (Pixel-based) WMs: Struggle with precise text rendering and discrete GUI dynamics. Existing visual WMs (e.g., VIMO) rely on complex, multi-stage pipelines involving external OCR, text masking, diffusion models, and closed-weight LLMs for text filling. This results in significant computational overhead, latency, and poor reproducibility due to closed weights.

Furthermore, existing benchmarks often simplify the problem by converting actions to text or testing only in-distribution scenarios, failing to assess zero-shot generalization or native visual action spaces.

Methodology

The authors propose gWorld, a novel paradigm for visual mobile GUI world modeling via renderable code generation. Instead of generating pixels directly, the model predicts the next GUI state as executable web code (HTML/CSS) that renders to pixels.

Core Architecture

Model: gWorld is built upon open-weight Vision-Language Models (VLMs), specifically fine-tuned versions of Qwen3 VL (8B and 32B parameters).
Representation: The model outputs structured web code. This leverages the VLM's linguistic priors for precise text rendering and its pre-training on structured web code for high-fidelity visual generation.
Training Objective: The model is trained to predict both a reasoning trace ( $R_t$ ) and the code-based next state ( $S^{code}_{t+1}$ ) given a current image state ( $S_t$ ) and action ( $A_t$ ).

Data Generation Framework

Since no code-based GUI world modeling datasets exist, the authors introduce a framework to synthesize training data:

Repurposing Trajectories: Offline mobile agent policy trajectories (from AitW, GUIOdyssey, AndroidControl, AMEX) are converted into transition triplets $(S_t, A_t, S_{t+1})$ .
Cross-modal Re-labeling: The ground-truth next state image ( $S^{image}_{t+1}$ ) is converted into renderable web code ( $S^{code}_{t+1}$ ) using a frontier model (Gemini 3 Flash) via an image-to-code prompt.
Look-ahead Reasoning: Reasoning traces ( $R_t$ ) are synthesized using the frontier model with "free look-ahead" access to the ground-truth next state. This decomposes the problem into predicting state changes in natural language first, then converting to code, ensuring alignment between reasoning and the final output.

Benchmark: MWMBENCH

The authors introduce MWMBENCH, a comprehensive benchmark consisting of 6 datasets:

In-Distribution (ID): 4 datasets (AitW, GUIOdyssey, AndroidControl, AMEX).
Out-of-Distribution (OOD): 2 new datasets:
- AndroidWorld: Automated collection from the AndroidWorld benchmark.
- KApps: Manually curated Korean-language trajectories featuring popular Korean apps (e.g., Baemin, KakaoTalk), testing multilingual and regional generalization.
Key Features: MWMBENCH evaluates models in the native visual modality (preserving original coordinates for actions) rather than converting actions to text, and includes OOD evaluation for zero-shot generalization.

Key Contributions

gWorld Models: The first open-weight, single self-contained visual mobile GUI WMs (8B and 32B) that operate via renderable code generation.
Data Generation Pipeline: A framework that automatically synthesizes large-scale, code-based training data with reasoning traces from existing policy trajectories.
MWMBENCH: A new benchmark suite enabling systematic evaluation of visual world modeling across in-distribution and out-of-distribution settings with native visual action spaces.
Empirical Validation: Demonstration that code-based modeling outperforms pixel-based generation and text-based abstraction, establishing a new Pareto frontier in accuracy versus model size.

Results

Extensive evaluations across 6 benchmarks (4 ID, 2 OOD) show:

Performance: gWorld 8B and 32B achieve the best and second-best Instruction Accuracy (IAcc.) across all benchmarks. They outperform 8 frontier open-weight models (including Llama 4 402B and Qwen3 VL 235B) that are up to 50.25× larger.
Pareto Frontier: gWorld sets a new Pareto frontier, achieving higher accuracy with significantly fewer parameters than existing state-of-the-art models.
Robustness: The models demonstrate strong generalization on OOD benchmarks (AndroidWorld and KApps), with performance not significantly degrading compared to ID settings.
Rendering Fidelity: The code-based approach virtually eliminates structural errors, with Render Fail rates below 1% (compared to >30% for some baselines).
Scaling Laws: Performance follows a predictable power law with dataset size (37K to 240K samples), suggesting further gains are possible with more data.
Downstream Impact: Integrating gWorld into a policy agent (M3A) via breadth-wise rollout and value estimation yields significant performance gains (e.g., +22.4% average accuracy improvement over baseline) compared to using the policy alone or a value function without a world model.
Efficiency: gWorld is substantially faster than prior pixel-based WMs (e.g., VIMO). End-to-end latency is 0.55s (8B) and 1.30s (32B), compared to VIMO's reported 160s.

Significance and Claims

The paper claims that renderable code world modeling offers a practical foundation for scalable and efficient mobile GUI agents. By shifting the representation from pixels to code, gWorld preserves GUI structure and text fidelity while avoiding the complexity and latency of multi-stage pixel-based pipelines.

The authors emphasize that:

This approach resolves the trade-off between visual fidelity and text precision.
It enables the creation of open-weight, reproducible world models that outperform much larger closed-weight or pixel-based alternatives.
The method is highly efficient, making real-time world modeling feasible for mobile agents.
The introduced benchmarks and data generation framework provide a necessary infrastructure for future research in mobile GUI world modeling.

The paper remains modest regarding limitations, noting that photo-realistic content (e.g., complex natural imagery in video players) is difficult to reconstruct with high fidelity via web code, though this does not significantly impact the semantic utility for most GUI tasks. Future work is suggested to incorporate explicit working memory for long-horizon dependencies.