Technical Summary: Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly utilized to translate webpage screenshots into executable front-end code. While these systems demonstrate strong capabilities in generating globally plausible HTML and CSS, this paper identifies a specific failure mode: visual pattern-completion bias.

The core problem is that when a localized visual deviation in a screenshot conflicts with a statistically dominant, repeated UI pattern (e.g., a single card that is wider than the others in a grid), MLLMs tend to ignore the visual evidence ("pixels") and default to the pattern-consistent value found in the surrounding context ("pattern"). This results in code that is structurally plausible but visually unfaithful to the source design, particularly for fine-grained details like font sizes or subtle spacing variations. Unlike general hallucination, this bias is systematic, driven by the model's tendency to complete patterns rather than ground predictions in specific visual inputs.

Methodology

Benchmark Construction (Pattern2Code)

The authors introduce Pattern2Code, the first benchmark specifically designed to measure visual pattern-completion bias in screenshot-to-code generation.

• Source Data: The benchmark is curated from 30 real-world webpages selected from the Design2Code dataset, filtered for clear repeated UI structures.
• Perturbation Strategy: For each webpage, the authors create controlled instances where exactly one element in a repeated pattern is perturbed. Two families of perturbations are studied:
  1. Structural Card Patterns: Modifying the inline CSS width of one card in a stacked panel.
  2. Text Style Patterns: Modifying the font-size of one text element in a sequence.
• Variables:
  • Positions: Perturbations occur at the first, middle, or last position of the pattern.
  • Magnitudes: Values are adjusted to 80%, 90%, 110%, or 120% of the baseline (100%).
  • Visual Conditions: Screenshots are rendered under Standard conditions and Noise-overlaid conditions (random opaque rectangles added to reduce visual saliency).
• Task Formulation: The model is presented with a screenshot and a masked HTML snippet (e.g., width: __ !important;). The task is a "fill-in-the-blank" objective: predict the missing percentage value to recreate the specific perturbed element.

Evaluation Setup

• Models: Five frontier proprietary MLLMs were evaluated:
  • Code-specialized: Codex-5.3, Opus-4.6.
  • General-purpose: ChatGPT-5.3, Sonnet-4.6, Flash-3.0.
• Metrics:
  • Accuracy: Exact recovery of the ground-truth perturbed value.
  • Bias Rate: The frequency of predicting the pattern-consistent baseline (100%) instead of the visual evidence.
  • Other Error: Predictions that deviate from the baseline but still miss the ground truth.

Key Results

1. Prevalence of Bias

All five evaluated models exhibit strong pattern-completion bias. When visual evidence conflicts with the repeated pattern, models overwhelmingly default to the pattern.

• Card Patterns (High Saliency): Mean bias rate is 69.78%. The best-performing model, Codex-5.3, achieves 68.61% accuracy, while others drop below 15% accuracy.
• Text Patterns (Low Saliency): Mean bias rate rises to 80.22%, with mean accuracy dropping to 7.89%. Flash-3.0 reaches a 96.11% bias rate on text, effectively collapsing to the baseline.

2. The Role of Visual Saliency

The severity of the bias is strongly correlated with the visual saliency of the deviation.

• Saliency Gradient: Coarse, layout-level deviations (card width) are detected more often than fine-grained, character-level deviations (font size).
• Noise Impact: Adding visual noise increases the bias rate (e.g., Flash-3.0's card bias increased by 9.44% under noise).
• Magnitude Impact: Subtler perturbations (e.g., 110% vs. 80%) significantly increase bias.
• Position Impact: Deviations in the middle of a pattern (flanked by consistent elements) are slightly easier to detect than those at the boundaries, though the effect is smaller than noise or magnitude.

3. Reasoning and Failure Modes

• Reasoning Effort: Longer reasoning chains correlate with lower bias rates. For ChatGPT-5.3, moving from zero reasoning tokens to intermediate effort reduced bias from 93.2% to ~40%. However, accuracy gains plateau, and increased effort does not guarantee correctness.
• Qualitative Analysis: Analysis of reasoning traces reveals a "Pattern Conformity Behavior":
  1. The model observes the visual deviation.
  2. It performs correct reasoning (e.g., estimating the width is ~118%).
  3. Override: It explicitly discards its own observation to conform to the pattern (e.g., "However, 100% seems most consistent with the pattern").
• Failure Modes: In 74% of biased responses, the model engaged with the screenshot. The dominant failure mode (59.1%) was "Perceived-Consistent" (the model failed to register the deviation despite seeing it), followed by "Observed-then-Overrode" (14.8%), where the model saw the difference but chose the pattern anyway.

Contributions

1. Problem Formulation: The paper defines visual pattern-completion bias as a distinct failure mode in screenshot-to-code generation, distinguishing it from general hallucination or poor global fidelity.
2. Benchmark: Introduction of Pattern2Code, a curated benchmark of 1,440 evaluated screenshots derived from 30 real-world webpages, designed to isolate localized visual deviations against repeated patterns.
3. Empirical Evidence: Demonstration that pattern-completion bias is strongly associated with visual saliency. The paper shows that even state-of-the-art models fail to preserve fine-grained design details (like font sizes) when they conflict with a dominant pattern.
4. Reasoning Analysis: Qualitative evidence showing that models often identify the correct visual anomaly but actively override their own reasoning to satisfy pattern consistency.

Significance and Claims

The paper claims that current screenshot-to-code systems cannot be trusted for fine-grained design fidelity without explicit verification.

• Trust Threshold: The authors argue that while these models may be reliable for coarse layout decisions (high-saliency features), they systematically fail on properties critical to polished front-end work (e.g., subtle spacing, font sizing) where visual saliency is low.
• Metric Limitations: The study highlights that standard end-to-end benchmarks (measuring global similarity) may hide these local grounding failures, as the generated code often looks globally plausible despite being locally incorrect.
• Implications for Practice: Developers using these tools should treat generated code as a draft that preserves global structure but may silently normalize fine-grained styling. The authors suggest that mitigation strategies (such as prompts encouraging element-by-element analysis or post-generation visual verification) are necessary to address this bias.

The work concludes that the tension between prior knowledge (pattern completion) and input evidence (visual grounding) remains a critical challenge for multimodal models in software engineering, requiring new evaluation frameworks that specifically target localized deviations.