Technical Summary: VCG-Bench

Problem Statement

Despite rapid advancements in Vision-Language Models (VLMs), a critical capability gap remains in handling structured, controllable diagrammatic tasks essential for professional workflows (e.g., flowcharts, system architectures, UI mockups). Existing approaches predominantly rely on "Diagram-as-Pixels" paradigms, utilizing rasterized generation and editing pipelines (e.g., diffusion models). These methods operate in probabilistic pixel spaces, leading to inherent limitations:

• Lack of Editability: Modifications often require regenerating the entire image, causing structural drift, blurry lines, text distortion, and hallucinated elements.
• Low Fidelity: They struggle to preserve precise semantics, logical topology, and vector-accurate outputs required for professional tools like Draw.io.
• Stochasticity: The generation process is inherently stochastic, making it difficult to ensure instruction compliance and deterministic outcomes.

Current benchmarks largely focus on single-domain generation (e.g., charts, scientific plots) or visual understanding, failing to address the dual requirements of generating editable structured code from images and editing that code based on natural language instructions.

Methodology

The authors propose a "Diagram-as-Code" paradigm, shifting from pixel manipulation to symbolic logic using mxGraph Extensible Markup Language (XML). This format represents diagrams as semantic entities (nodes and edges) with explicit topological and geometric constraints, enabling precise, low-cost, and deterministic modifications.

VCG-Bench Framework

VCG-Bench is a unified benchmark designed around a "Data-Task-Evaluation" framework:

1. Dataset Construction

• Scale & Diversity: A taxonomized dataset of 1,449 diverse diagrams spanning 6 major domains (Academic, Software, Business, Management, UI/UX, General) and 15 sub-domains.
• Sources: Data is curated from open-source repositories (40%), open-access academic papers (30%), anonymized corporate diagrams (20%), and permissively licensed web diagrams (10%).
• Quality Assurance: A "Machine-Generation, Human-Verification" pipeline ensures structural completeness, text accuracy (>95% OCR match), and rendering fidelity (cosine similarity >0.85 via SigLIP2).
• Difficulty Stratification: Samples are stratified by complexity (Easy, Medium, Hard) based on token counts for generation and the number of atomic operations for editing.

2. Task Definitions
The benchmark formalizes two complementary tasks requiring valid, parsable mxGraph XML output:

• Task 1: Vision-to-Code Generation: Given a raster image and a structured caption, the model generates a valid mxGraph XML string that approximates the source in structure, semantics, and layout.
• Task 2: Instruction-to-Patch Editing: Given a source mxGraph XML, its rendering, and a natural language instruction, the model predicts a Differential Patch (a JSON-based replacement of original/modified XML fragments). This employs an Incremental Modification Strategy to avoid token-expensive full regeneration and structural drift.

3. Evaluation Protocol
The authors introduce a multi-dimensional evaluation suite beyond simple text matching:

• Executability: Execution Success Rate (ESR) measures syntactic validity and renderability.
• Visual Fidelity:
  • Style Consistency Score (SCS): A VLM-based metric (0-10 scale) evaluating visual style, layout consistency, and aesthetic quality.
  • SigLIP2 Visual Similarity: Cosine similarity between embeddings of rendered images to capture high-level structural alignment.
• Semantic Compliance:
  • CodeXQA (Task 1): Evaluates topological preservation via question-answering on the generated XML (Counting, Identification, Relationship).
  • XDRFR (Task 2): XML-based Decomposed Requirements Following Ratio. Instructions are decomposed into atomic Yes/No verification questions checked directly against the modified XML structure, avoiding visual ambiguities.

Key Contributions

1. VCG-Bench: The first unified benchmark bridging the gap between vision and editable structured generation, specifically targeting the mxGraph XML format used in professional diagramming tools.
2. High-Quality Taxonomized Dataset: A curated collection of 1,449 diagrams with provenance, structural diversity, and explicit difficulty stratification across multiple domains.
3. Comprehensive Evaluation Protocol: Novel metrics (ESR, SCS, CodeXQA, XDRFR) that jointly assess executability, style consistency, and instruction following, enabling rigorous benchmarking of State-of-the-Art (SOTA) VLMs.
4. Paradigm Shift: Demonstrates the efficacy of the "Diagram-as-Code" approach over "Diagram-as-Pixels," showing that symbolic representation enables precise, localized edits that are infeasible in pixel space.

Experimental Results

The benchmark evaluated leading closed-source (Gemini-3-Pro, GPT-5.2, Claude-4.5) and open-source (Qwen3-VL series, GLM-4.6V, etc.) models.

• Vision-to-Code (Task 1):

  • Closed-Source Dominance: Closed-source models (e.g., Gemini-3-Pro) achieved high ESR (>0.96) and strong perceptual scores.
  • Open-Source Limitations: Open-source models frequently failed on executability (ESR often 0.00–0.35), despite moderate semantic understanding (CodeXQA). The primary failure mode was instance grounding (counting elements), leading to topologically incorrect graphs.
  • Difficulty Sensitivity: Performance on open-source models degraded sharply as task difficulty increased, whereas closed-source models showed controlled degradation.

• Instruction-Based Editing (Task 2):

  • High Fidelity: When provided with structured code, all models (both open and closed) achieved near-perfect ESR (>0.99) and high SCS.
  • Instruction Following: XDRFR became the key discriminative metric. Gemini-3-Pro led with 0.9405, while open-source models like Qwen3-VL-235B reached 0.9312, narrowing the gap seen in Task 1.
  • Paradigm Validation: The results confirm that once a diagram is represented as code, constrained incremental editing is a robust and precise paradigm, significantly outperforming pixel-based editing baselines (which dropped to 0.649 DRFR on hard tasks).

Significance and Claims

The paper claims that VCG-Bench exposes significant limitations in current SOTA VLMs regarding fine-grained visual parsing and syntactic constraint satisfaction in professional engineering workflows.

• Separation of Capabilities: The benchmark reveals a sharp gap between "vision-to-code" (generating structure from pixels) and "code-to-code" (editing structure). Models often fail to infer structure from images but can effectively edit provided code.
• Professional Readiness: By shifting focus from passive perception to active, editable code generation, VCG-Bench provides a reproducible testbed to assess the practical readiness of multimodal agents for real-world diagramming tasks.
• Data Engine: Beyond evaluation, the framework serves as a scalable data engine to generate high-quality, verified visual-code pairs, addressing the scarcity of structured diagrammatic data for training VLMs.

The authors emphasize that while the "Diagram-as-Code" paradigm offers superior controllability and editability, current models still struggle with the initial translation from unstructured pixels to structured symbolic representations, particularly in complex, logic-intensive diagrams.