Technical Summary: ChartSync

Problem Statement

Generative image editing models have advanced rapidly in natural scenes but struggle significantly when applied to structured statistical charts. The core challenge lies in Visuo-Logical Cascading Editing (VLCE): the requirement that modifying a textual data value (e.g., changing a percentage label) must trigger a synchronized, geometric adjustment of the corresponding visual elements (e.g., resizing a pie slice or bar height) based on underlying data dependencies.

Existing approaches face two primary limitations:

1. Localized Substitution: Most instruction-based editing models perform only literal text substitution, failing to propagate changes to the chart's geometry, resulting in visual inconsistencies where text and data no longer match.
2. Code-Paradigm Constraints: Recent chart editing works often rely on regenerating code from images. However, this assumes the availability of source code or suffers from information loss during the reverse-engineering of flattened chart images, making it unsuitable for real-world scenarios where only the rendered image is available.

Current benchmarks primarily evaluate text manipulation, target localization, or code-level correctness, but none explicitly isolate value-to-geometry synchronization as the core evaluation target.

Methodology

1. The ChartSync Benchmark

To address these gaps, the authors introduce ChartSync, an expert-validated benchmark designed to evaluate VLCE capabilities.

• Data Construction: The dataset comprises 870 expert-validated triplets (Original Image, Instruction, Ground Truth) across 9 chart categories (e.g., bar, pie, radar, PIP) and 4 task types.
• Programmatic Rendering Pipeline: To ensure deterministic visuo-logical coupling for the Ground Truth (GT), the authors utilize a three-stage pipeline:
  1. Semantic Instruction Synthesis: A VLM proposes instructions that implicitly require cascading updates (e.g., changing a value without explicitly stating the geometric change).
  2. Code-Driven Generation: An LLM modifies the original Python plotting scripts to reflect the instruction, triggering causal structural updates. A Logic Validator ensures syntax and semantic consistency.
  3. Expert Quality Assurance: Human experts review the triplets to ensure unambiguous instructions and unique, artifact-free GT images.
• Task Distribution: The dataset includes 235 geometry-coupled VLCE instances specifically designed to test dependency-aware cascading reasoning, alongside text-only tasks for baseline comprehension.

2. Evaluation Framework

ChartSync employs a two-tier evaluation framework:

• Objective Visual Metrics:
  • OCR F1: Measures textual content consistency at the character level.
  • SSIM (Structural Similarity Index): Assesses global structural integrity and non-edited region preservation.
• VLM-as-a-Judge Paradigm: A proprietary Vision-Language Model evaluates complex multimodal reasoning via three discrete scores:
  • TESR (Textual Edit Success Rate): Evaluates the semantic correctness and spatial precision of the edited text.
  • VLCS (Visuo-Logical Consistency Score): Specifically measures whether geometric regions coupled to the edited text correctly reflect the data modification.
  • BFS (Background Fidelity Score): Assesses the preservation of untargeted chart components to detect "bleeding" or collateral damage.

Key Results

The authors evaluated 14 instruction-based image editing models (both open-source and proprietary) and one code-mediated pipeline on ChartSync.

• Severe Failure in Cascading Updates: Most models exhibit a sharp performance drop when transitioning from text substitution to geometric synchronization. For instance, while some models achieve high Textual Edit Success (TESR), their Visuo-Logical Consistency (VLCS) scores plummet (e.g., Qwen-Image-Edit-2511: TESR 61.81 vs. VLCS 13.83).
• Code-Mediated Limitations: The code-mediated baseline achieved high text scores (OCR F1: 88.76) but suffered significant drops in structural fidelity (SSIM: 54.83, VLCS: 28.51), confirming that image-to-code reconstruction introduces substantial information loss.
• Emerging Capabilities in Frontier Models: Only two proprietary frontier models (Nano Banana Pro and GPT-Image-2) demonstrated strong VLCE capabilities, achieving high VLCS scores (83.71 and 74.47, respectively). However, even these models showed residual errors in semantic isolation (Background Fidelity).
• Error Analysis: The study identifies three primary failure modes:
  1. Textual Manipulation Failure (TMF): Inability to locate or render target text.
  2. Visuo-Logical Inconsistency (VLI): Failure to propagate text changes to geometry.
  3. Background Corruption (BC): Unintended modification of untargeted regions.
     Open-source models predominantly failed on TMF and VLI, while leading proprietary models primarily struggled with BC.

Significance and Contributions

The paper positions ChartSync as a critical diagnostic tool for advancing multimodal architectures in structured document understanding. Its main contributions are:

1. Formal Definition of VLCE: The authors formally define Visuo-Logical Cascading Editing, distinguishing it from generic image manipulation by emphasizing the strict dependency between data values and geometric primitives.
2. A Deterministic Benchmark: By using a programmatic rendering pipeline, ChartSync guarantees that the Ground Truth possesses perfect visuo-logical coupling, providing a reliable standard for evaluating geometric synchronization.
3. Comprehensive Evaluation Protocol: The integration of objective metrics with a VLM-as-a-Judge paradigm allows for the simultaneous assessment of low-level fidelity and high-level multimodal reasoning.
4. Identification of Meta-Abilities: Through error analysis, the authors deconstruct the requirements for robust chart editing into a hierarchy of meta-abilities: foundation perception (semantic grounding), synchronization and reasoning (cascading updates), and high-fidelity generation (semantic isolation).

The authors conclude that while frontier proprietary models show emerging VLCE capabilities, the field remains far from solved, particularly regarding the precise geometric synchronization required for real-world chart editing. ChartSync is released to guide the development of future models capable of handling these structured transformations.