🤖 AI

ControlGUI: Guiding Generative GUI Exploration through Perceptual Visual Flow

This paper introduces ControlGUI, a diffusion-based model that facilitates rapid low-effort exploration of diverse low-fidelity interface sketches by allowing designers to flexibly combine prompts, wireframes, and visual flows as input specifications.

Original authors: Aryan Garg, Yue Jiang, Antti Oulasvirta

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aryan Garg, Yue Jiang, Antti Oulasvirta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect trying to design a new house. In the old days, you had to draw every single brick, choose the exact shade of paint, and pick the furniture before you could even show your ideas to a client. If you wanted to try a different style, you had to erase everything and start over. This is how designing computer interfaces (like websites and apps) used to be: slow, rigid, and exhausting.

Then came AI tools that could generate ideas from text. You'd type "a cool cocktail bar website," and the AI would spit out a picture. But here's the problem: the AI is like a very enthusiastic but slightly confused intern. It might get the vibe right (lots of neon lights), but it often messes up the structure (putting the menu where the bathroom should be) or the flow (making you look at the footer before the main drink).

Enter ControlGUI. Think of it as a "Super-Intern" that doesn't just listen to your words, but also understands your sketches and even your eye movements.

Here is how it works, broken down into simple concepts:

1. The Three Magic Inputs

Most AI tools only listen to one thing: your text prompt. ControlGUI is special because it can listen to three different languages at the same time, and you can mix and match them however you like:

The Whisper (Text Prompt): You say, "I want a website for a toy store." This sets the mood and the theme.
The Skeleton (Wireframe): You draw a rough sketch with boxes. "Here is where the big image goes, here is the button, here is the text." This tells the AI, "Don't mess up the layout; keep the structure I drew."
The Eye-Track (Visual Flow): This is the secret sauce. You draw a line showing the order in which you want a user to look at things. "First, they should see the toy, then the price, then the 'Buy' button." This guides the AI on how to arrange the design so it feels natural to the human eye.

The Analogy: Imagine you are directing a movie.

Text Prompt is the script ("It's a scary horror movie").
Wireframe is the storyboard (showing where the actors stand).
Visual Flow is the camera direction (telling the audience exactly where to look first to build suspense).
ControlGUI lets you give the director all three instructions at once, ensuring the final scene looks exactly how you imagined it.

2. The "Dual-Adapter" Brain

How does the computer actually do this? The researchers built a special brain for the AI called a Dual-Adapter Diffusion Model.

Think of the AI's brain as a massive library of every website ever made.

Adapter A (The Architect): This part of the brain is obsessed with structure. It looks at your wireframe and says, "Okay, the button must be here. The image must be there." It ensures the building doesn't collapse.
Adapter B (The Psychologist): This part of the brain is obsessed with human behavior. It looks at your "Visual Flow" and says, "Humans look at the top-left first, then the center. Let's arrange the colors and shapes to guide their eyes exactly like that."

By having these two "specialists" working together, the AI doesn't just make a pretty picture; it makes a functional design that follows your rules.

3. The Dataset: The "Training Gym"

To teach this AI, the researchers didn't just use random pictures. They built a massive gym called a Dataset with 72,500 examples.

They took real websites and apps.
They stripped them down to their "skeletons" (wireframes).
They wrote descriptions of what they were.
Crucially, they used AI to predict how a human's eyes would move across the screen (scanpaths).

It's like training a chef not just on recipes, but on how people actually eat the food. The AI learned that if you put a giant "Sale" sign in the middle, people will look there first.

4. Why This Matters (The User Study)

The researchers tested this with real designers. They found that when designers used ControlGUI:

They were faster: They could generate 100 different ideas in under two minutes.
They were more creative: Because the AI handled the boring details (like "put the button here"), the designers could focus on the big ideas.
They felt more in control: Instead of fighting the AI to get the layout right, they could just draw a rough box, and the AI would respect it.

The Catch (Limitations)

It's not perfect yet. The paper admits that the text generated by the AI can sometimes look like gibberish (like "Lorem Ipsum" but worse), and the images are a bit low-resolution. It's like a rough sketch, not a finished painting. But for the early stage of "brainstorming," that's exactly what you need. You don't want a perfect painting when you're just trying to figure out if the kitchen should be on the left or the right.

The Bottom Line

ControlGUI is a tool that turns the chaotic process of "guessing what the AI will make" into a collaborative conversation. It respects your rough sketches, listens to your text, and understands how humans actually look at screens. It's not about replacing designers; it's about giving them a superpower to explore a thousand ideas in the time it used to take to draw one.

1. Problem Statement

Early-stage Graphical User Interface (GUI) design requires exploring a wide range of low-fidelity concepts to avoid premature commitment to specific ideas. However, existing tools face significant limitations:

Constraint vs. Flexibility: Sketching tools often anchor designers to initial ideas (fixation), while constraint-based systems require overly detailed, time-consuming specifications that designers cannot provide early in the process.
Limitations of Current Generative AI: Most existing text-to-GUI models rely solely on natural language prompts. They struggle to preserve specific structural layouts (wireframes) and, crucially, fail to account for visual flow (the sequence in which users attend to interface elements), which is vital for usability but difficult to specify via text.
Data Gap: There is a lack of large-scale, multimodal datasets that align GUI screenshots with wireframes, textual descriptions, and user attention data (scanpaths).

2. Methodology

The authors propose ControlGUI, a controllable diffusion model designed to generate diverse, low-fidelity GUIs based on flexible combinations of three input modalities: Text Prompts, Wireframes, and Visual Flows.

A. Dataset Construction

The authors created a new large-scale multimodal dataset containing 72,500 GUIs (combining mobile apps and webpages).

Sources: Aggregated and cleaned from existing datasets (ENRICO, VINS, WebUI).
Annotations: Each sample includes:
- High-resolution screenshots.
- Wireframes: Labeled bounding boxes for 9 element types (e.g., Button, Text, Image).
- Descriptions: Generated using LLaVA-Next (Vision-Language Model).
- Scanpaths: Predicted user eye-tracking paths generated using EyeFormer (a state-of-the-art scanpath prediction model), serving as a proxy for visual flow.

B. Model Architecture: Dual-Adapter Diffusion

Built upon Stable Diffusion, the model employs a dual-adapter architecture to simultaneously control local structure and global perceptual flow:

Wireframe Adapter (Local Control):
- Inspired by ControlNet, this adapter uses a trainable copy of the encoder and middle blocks with zero-initialized convolutions.
- It ensures structural fidelity, aligning the generated GUI's element placement and types with the input wireframe.
Flow Adapter (Global Control):
- Utilizes additional cross-attention layers within the diffusion model.
- It encodes visual flow specifications (either a reference GUI or a specific element order) using embeddings from EyeFormer.
- It guides the perceptual flow, ensuring the generated design directs user attention in the intended sequence.

C. Training and Optimization

Classifier-Free Guidance (CFG): Used to balance adherence to input conditions with output diversity.
Loss Functions:
- CFG Loss: Standard diffusion loss for both adapters.
- Flow Consistency Loss ( $L_{flow}$ ): A differentiable Soft-DTW (Dynamic Time Warping) loss. Since ground-truth scanpaths are unavailable for generated images, the model uses EyeFormer to predict the scanpath of the generated GUI and minimizes the distance between this predicted path and the target flow specification.
Training Strategy: Three-stage training (Wireframe adapter, Flow adapter, then Joint Fine-tuning) using DDIM sampling for reproducibility.

D. Post-Processing: Text Enhancement

To address the "text problem" common in diffusion models (gibberish or unreadable text), the authors implemented a Vision-Language-guided text correction pipeline. It uses LLaVA-Next to detect illegible text regions and replace them with contextually appropriate, readable text via inpainting.

3. Key Contributions

ControlGUI System: The first diffusion-based model for GUI generation that supports flexible, combinable inputs of text, wireframes, and visual flow.
Dual-Adapter Architecture: A novel mechanism that decouples local structural control (wireframes) from global perceptual control (visual flow), enabling alignment with both layout and user attention intent.
Large-Scale Multimodal Dataset: A dataset of 72,500 GUIs with aligned screenshots, wireframes, descriptions, and predicted scanpaths, filling a critical gap in GUI research resources.
Text Correction Pipeline: An automated method to enhance the realism and usability of generated GUI text without designer intervention.

4. Results and Evaluation

Quantitative Evaluation

Baselines: Compared against ControlNet, IP-Adapter, Stitch, and GPT-5.
Metrics:
- Perceptual Quality (ManIQA, MUSIQ): ControlGUI achieved the highest scores (0.61 ManIQA), outperforming baselines.
- Prompt Alignment (CLIP-IQA): Superior semantic alignment (0.53).
- Flow Alignment (Soft-DTW): ControlGUI was the only method capable of modeling visual flow, achieving a score of 0.55.
Efficiency: Generates a batch of 16 diverse GUIs in ~19 seconds on a single RTX 4090 GPU.

Qualitative Evaluation

Demonstrated that adding wireframes enforces structural consistency without sacrificing stylistic diversity.
Visual flow inputs successfully guided the generated attention trajectories to match specified element orders.

User Study

A within-subjects study with 18 designers evaluated the system against a prompt-only baseline.

Diversity: Participants rated ControlGUI significantly higher for design diversity (4.59 vs. 3.56).
Control & Intent: Multimodal inputs significantly improved perceived control (4.00 vs. 2.58) and alignment with design intent.
Creativity: The Creative Support Index (CSI) was significantly higher (51.31 vs. 31.11), indicating the system acts as a catalyst for divergent thinking and idea validation.
Limitations Noted: Participants cited low resolution, unstable text generation (before correction), and difficulty with highly specific content (e.g., specific product images) as current limitations.

5. Significance

ControlGUI represents a significant shift in Generative AI for HCI.

Bridging the Gap: It moves beyond simple text-to-image generation to a multimodal, intent-aware system that respects both the structure (wireframe) and the experience (visual flow) of a design.
Design Workflow Integration: By allowing designers to provide partial or underspecified inputs (e.g., just a rough sketch or a specific attention order), it supports the natural, iterative nature of early-stage ideation without forcing over-specification.
Perceptual Design: It is the first system to explicitly integrate visual flow (user attention) as a controllable generation parameter, addressing a critical usability factor that is typically only evaluated late in the design process via expensive user studies.

In conclusion, ControlGUI provides a flexible, rapid, and effective tool for designers to explore a broad design space, generating realistic, diverse, and intent-aligned GUI concepts that serve as a strong foundation for further high-fidelity development.