Technical Summary: Vision2Code

Problem Statement

Image-to-code generation evaluates whether a Vision-Language Model (VLM) can reconstruct the structure of an image into executable code. Unlike pixel-based generation, code output is editable, verifiable, and reusable. However, existing benchmarks suffer from three primary limitations:

1. Narrow Scope: Most benchmarks focus on single visual domains (e.g., only charts or only web UIs).
2. Dependency on Reference Code: Many require paired executable reference code for evaluation, limiting applicability to datasets where source programs are unavailable (e.g., scanned documents, textbook diagrams).
3. Generic Evaluation: Existing metrics often rely on generic rubrics or embedding similarities that fail to capture domain-specific reconstruction errors (e.g., incorrect chemical bonds, collapsed 3D depth, or wrong graph topology).

Methodology

The authors introduce Vision2Code, a reference-code-free benchmark and evaluation framework designed to assess image-to-code generation across 15 source datasets spanning six visual domains: charts/plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes.

Benchmark Construction

• Scale: The benchmark contains 2,169 test examples (plus a 539-example "test-mini" subset for lower-cost evaluation).
• Domains: The six domains are selected to expose distinct reconstruction challenges:
  • Charts/Plots: Axis scales, tick spacing, legend binding.
  • Geometry: Vertex/angle labeling and spatial adjacency.
  • Graphs: Node identity, edge connectivity, and directionality.
  • Scientific Imagery: Specialized notation (chemistry, physics, circuits).
  • Documents: Dense OCR, column structure, and reading order.
  • 3D Spatial Scenes: Object identity, depth ordering, and spatial relations.
• Data Source: Examples are drawn from public datasets (e.g., ChartQA, DocVQA, ChemVQA-2K) where original rendering programs are often unavailable.

Evaluation Framework

Instead of comparing generated code to a reference program, Vision2Code evaluates the rendered output of the generated code against the source image.

1. Execution: Generated Python code (using Matplotlib/NumPy by default) is executed in a sandboxed environment. Code that fails to execute, times out, or produces no valid image receives a score of 0.0.
2. VLM Rater: Successful renders are scored by a VLM rater (Qwen3.5-122B) using dataset-specific rubrics.
   • Each dataset has a unique rubric with four weighted categories (summing to 1.0).
   • Scores are assigned on a 0–5 scale.
   • Guardrails: Deterministic caps are applied to prevent high scores for severe semantic failures (e.g., wrong molecular structure, flattened 3D scenes, or incorrect chart orientation).
3. Diagnostics: The framework separates "render success" (code execution) from "reconstruction quality" (visual fidelity).

Validation

The authors validate the evaluation protocol against human judgments and automatic baselines:

• Human Alignment: The dataset-specific rubric achieves a Pearson correlation of 0.723 and Spearman of 0.703 with human ratings.
• Baselines: This outperforms a generic rubric (0.626/0.633) and embedding-similarity baselines (0.407/0.498), demonstrating that domain-specific criteria are essential for accurate evaluation.

Key Results

The authors evaluated nine models (five open-weight, four proprietary) on the benchmark.

• Overall Performance: The best model, GPT-5.4 Mini with reasoning, achieved a macro-average score of 4.06/5.0 on the full test set. Even top-tier models show significant room for improvement.
• Domain Dependence: Performance varies drastically by domain.
  • Models excel on regular charts and graphs (e.g., GPT-5.4 scores ~4.7 on DVQA and GraphVQA).
  • Models struggle with spatial and symbolic domains. The best score on the 3D SpatialVLM-QA dataset is only 2.42/5.0, and chemistry (ChemVQA-2K) scores drop to ~0.74 for some models.
• Reasoning: Enabling reasoning capabilities improves scores across all models (e.g., Qwen3.5-397B improves from 2.27 to 2.95), but does not fully resolve failures in the most complex domains (spatial, chemistry).
• Error Analysis:
  • Execution Failures: Vary by model; smaller models often fail due to syntax/truncation, while larger models fail due to hallucinated APIs or shape mismatches.
  • Reconstruction Errors: Once code executes, errors shift to semantic content (e.g., missing text, wrong topology).
• Self-Training: Using the evaluator to filter model outputs for self-training improves performance. Fine-tuning Qwen3.5-9B on "rater-filtered" data (where the model reconstructs well once but inconsistently) improved its score from 1.60 to 1.86 without paired source code.
• Test-Time Scaling: Repeated refinement at inference time yields limited gains and can introduce semantic drift.

Significance and Claims

The paper claims that Vision2Code provides a reproducible testbed for measuring, diagnosing, and improving image-to-code generation in settings where paired reference code is unavailable. Its primary contributions are:

1. A Multi-Domain Benchmark: Enabling evaluation across 15 datasets and 6 visual domains, moving beyond the narrow scope of previous chart-only or UI-only benchmarks.
2. A Domain-Aware Evaluation Framework: Demonstrating that dataset-specific rubrics and deterministic guardrails are necessary to align automatic evaluation with human judgment, particularly for detecting domain-specific semantic errors.
3. A Training Signal Source: Showing that evaluator-filtered outputs can serve as effective supervision for improving VLM capabilities without requiring ground-truth code.

The authors conclude that while current VLMs can reconstruct regular visual primitives, they remain weak on tasks requiring precise preservation of domain-specific structures (spatial relations, chemical symbols, circuit topology). The benchmark highlights that high aggregate scores can mask significant failures in specific, critical domains.