DOne: Decoupling Structure and Rendering for High-Fidelity Design-to-Code Generation

The paper introduces DOne, a framework that decouples structure understanding from element rendering to overcome the holistic limitations of Vision Language Models in Design-to-Code generation, achieving superior visual fidelity and productivity on the new HiFi2Code benchmark.

The Gist

Imagine you are a master architect who has drawn a beautiful, intricate blueprint for a new skyscraper. You hand this blueprint to a robot builder and say, "Build this exactly as I drew it."

In the world of web development, this is the challenge of Design-to-Code: turning a visual picture of a website into the actual code (HTML/CSS) that makes it work.

For a long time, the robots (AI models) trying to do this had a major problem. They were like a builder who looked at the whole skyscraper at once and got overwhelmed. They would guess the general shape but miss the details. They might build the walls in the wrong place, forget the windows, or replace a fancy glass door with a generic cardboard box.

The paper you shared introduces a new system called DOne (Decoupling Structure and Rendering) to fix this. Here is how it works, explained with simple analogies:

1. The Problem: The "Holistic Bottleneck"

Imagine trying to describe a complex painting to a friend by looking at the whole thing at once. You might say, "It's a house with trees," but you'd forget the specific color of the front door or the pattern on the roof tiles.

Current AI models suffer from this "Holistic Bottleneck." They try to understand the entire website layout and generate the code in one giant leap. Because the task is too big, they get confused, leading to:

• Broken Layouts: The menu ends up at the bottom of the page.
• Missing Details: Logos and icons disappear or turn into gray boxes.
• Generic Replacements: Instead of a specific product image, they just put a placeholder that says "Image."

2. The Solution: DOne's "Three-Step Chef" Approach

The authors realized that to build a perfect website, you shouldn't try to do everything at once. Instead, you should break the job down into three specialized steps, like a high-end restaurant kitchen.

Step 1: The "Floor Planner" (Layout Segmentation)

Before the chef starts cooking, the floor planner looks at the restaurant and draws lines to separate the kitchen, the dining area, and the bar.

• What DOne does: Instead of looking at the whole website as one messy image, DOne uses a smart AI to slice the design into logical chunks (Header, Sidebar, Main Content, Footer).
• The Analogy: It's like cutting a complex pizza into slices before eating it. This prevents the AI from getting overwhelmed by the whole pie at once.

Step 2: The "Inventory Manager" (Element Retrieval)

Once the slices are defined, the Inventory Manager goes through the kitchen and grabs every single specific ingredient: the fresh basil, the specific brand of cheese, the unique sauce.

• What DOne does: It has a special tool that hunts down tiny, specific details like icons, logos, and buttons. It doesn't just guess; it finds the exact image file and saves it.
• The Analogy: If the design has a tiny red heart icon, the Inventory Manager finds that exact heart and puts it in a box, ensuring it doesn't get lost or replaced with a generic "heart" emoji.

Step 3: The "Architect's Blueprint" (Schema-Guided Generation)

Now, the builder (the AI that writes the code) doesn't just guess. They are handed a strict, step-by-step blueprint (a JSON schema) that says: "Put the Header here, put the Red Heart icon inside the Header, and make sure the Sidebar is 200 pixels wide."

• What DOne does: It creates a logical map of the website first. Then, it tells the code-writing AI to follow this map strictly.
• The Analogy: Instead of telling the builder, "Build a house," you give them a detailed instruction manual: "First, build the foundation. Then, place the front door at coordinate X. Then, hang the specific red heart icon on the door."

3. The Result: A "High-Fidelity" Website

Because DOne separates the structure (where things go) from the rendering (what things look like), the final result is amazing.

• Before: The AI might build a house that looks like a house, but the door is on the roof, and the windows are missing.
• With DOne: The house looks exactly like the blueprint. The door is in the right place, the windows are clear, and the red heart icon is perfectly placed.

Why Does This Matter?

The authors also created a new "test" called HiFi2Code. Think of this as a super-hard driving test for AI. Previous tests were like driving in an empty parking lot; this new test is like driving in rush hour traffic with complex intersections.

When they tested DOne on this hard course:

• It was 3 times faster for human developers to get a working website.
• The websites looked much more like the original designs (over 10% better than the best previous methods).

Summary

DOne is like hiring a team of specialists instead of one overworked generalist.

1. One person slices the design into manageable pieces.
2. One person collects all the tiny details.
3. One person assembles the code based on a strict map.

By doing this, the AI stops guessing and starts building with precision, turning a blurry sketch into a pixel-perfect website.

Technical Summary

1. Problem Statement

The paper addresses the "holistic bottleneck" currently plaguing Vision Language Models (VLMs) in Design-to-Code (Design2Code) generation. While VLMs have improved the automation of converting visual designs (screenshots/mockups) into executable HTML/CSS, they struggle to balance high-level structural hierarchy with fine-grained visual details.

The authors identify three specific failure modes in existing state-of-the-art methods (including heuristic cropping and direct generation approaches):

1. Erroneous Hierarchy Parsing: Models often misinterpret semantic relationships (e.g., confusing sidebars with content blocks), leading to flawed document structures.
2. Loss of Fine-Grained Visual Elements: Critical assets like icons, logos, and specific buttons are often replaced by generic placeholders or omitted entirely.
3. Holistic Processing Overload: Processing complex, dense webpages as a single monolithic image exceeds VLM capacity, forcing a trade-off where structural integrity is sacrificed for local details or vice versa.

2. Methodology: The DOne Framework

To overcome these limitations, the authors propose DOne, an end-to-end framework that decouples structural understanding from element rendering. The framework operates in three distinct stages:

A. Learned Layout Segmentation

Instead of relying on heuristic cropping, DOne employs a specialized learned layout segmentation model (based on RT-DETR) trained on a new dataset called WebSeg.

• Function: It decomposes a full-page screenshot into semantically meaningful, non-overlapping regions (e.g., headers, sidebars, main content).
• Optimization: A post-processing algorithm optimizes bounding boxes to resolve overlaps and ensure structural coherence, mimicking the macroscopic planning phase of human designers.

B. Fine-Grained Visual Element Retrieval

To handle the unique challenges of UI elements (extreme aspect ratios, high density, and semantic gaps), DOne uses a hybrid detection strategy:

• DETR: Used for large, context-dependent structures (e.g., Navbars, Hero images) to resolve global layout context.
• YOLOv10: Used for small, dense elements (e.g., icons, buttons) to ensure efficient detection of high-density components.
• Output: Detected elements are cropped and stored in a repository to be retrieved during code generation, preventing the loss of visual fidelity.

C. Schema-Guided Generation

DOne introduces a schema-guided generation paradigm to bridge the gap between visual layout and code:

1. Intermediate Representation: The system first infers a hierarchical JSON blueprint (Layout Schema) from the segmented layout and detected elements. This acts as a spatial-to-logical bridge.
2. Schema-Constrained Synthesis: The VLM is then tasked with "fleshing out" this schema into executable code. The schema explicitly grounds visual elements within their parent containers, ensuring the VLM focuses on styling and asset integration rather than inferring the layout structure from scratch. This prevents "structural hallucinations."

3. Key Contributions

The paper makes three primary contributions:

1. DOne Framework: A novel hierarchical framework unifying learned layout segmentation, hybrid element retrieval, and schema-guided generation to achieve high-fidelity reconstruction.
2. New Datasets:
   • WebSeg: A large-scale training dataset (30k images) with detailed layout annotations to train the segmentation model.
   • HiFi2Code: A rigorous evaluation benchmark (200 pages) featuring significantly higher layout complexity, element density, and structural nesting than existing datasets (e.g., Design2Code, Web2Code).
3. Performance Validation: Extensive experiments demonstrating that DOne significantly outperforms existing methods in both visual fidelity and developer productivity.

4. Experimental Results

The authors evaluated DOne on the HiFi2Code benchmark against baselines like S2C, CoT2C, and DCGen using various VLM backbones (Claude-3.7-Sonnet, Gemini-2.5-Pro, Qwen-VL-Max).

• High-Level Similarity: DOne achieved a CLIP Score of 0.7435 and a GPT Score of 0.723 (using Claude-3.7-Sonnet), outperforming the previous best (DCGen) by over 10% in GPT Score.
• Fine-Grained Metrics: DOne showed superior performance in Color Match (0.461 vs. <0.07 for baselines) and Text Match, proving it retrieves actual visual assets rather than wireframes.
• Human Evaluation:
  • Visual Fidelity: In a blind preference study, 58.6% of developers and designers preferred DOne over other methods.
  • Productivity: A controlled user study showed a 3× improvement in productivity, reducing average task completion time from 9.81 minutes to 3.18 minutes while maintaining higher visual fidelity.

5. Significance

The significance of this work lies in its shift from monolithic generation to modular, decoupled processing. By separating the "where" (structure/segmentation) from the "what" (elements/retrieval) and the "how" (schema-guided synthesis), DOne solves the fundamental trade-off between global layout accuracy and local detail preservation.

The release of HiFi2Code sets a new standard for evaluating Design2Code systems, moving beyond simple synthetic datasets to complex, real-world web interfaces. This framework not only improves the quality of automated code generation but also offers a practical tool to significantly accelerate the workflow of frontend developers.