ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in single-turn chart generation (creating a visualization from scratch based on a prompt), their ability to support real-world exploratory data analysis remains underexplored. In practice, data analysts and developers rarely create visualizations in a single step; instead, they engage in iterative, multi-turn workflows where they refine existing specifications, adjust encodings, add overlays, and reformat layouts based on evolving feedback.

Current benchmarks fail to capture this reality because they focus on one-shot generation or static question answering. Furthermore, existing evaluation methods (e.g., LLM-as-a-Judge or CLIP-based similarity) suffer from high variance, lack of interpretability, and an inability to detect semantically meaningful but visually subtle errors (e.g., incorrect data aggregation). There is a critical need for a benchmark that evaluates sustained, context-aware editing where models must maintain "common ground" across multiple turns while tracking prior edits.

2. Methodology: ChartEditBench

The authors introduce ChartEditBench, a synthetic benchmark designed to evaluate incremental, visually grounded chart editing via code modification.

A. Task Definition

The benchmark defines two complementary tasks, both evaluated over multi-turn conversations (typically 5 modification turns per instance):

1. Code-to-Code Modification: Given an original code-image pair ($c_1, p_1$) and a target modified image ($p_2$), the model must predict the modified code ($c_2$) that generates $p_2$.
2. Natural Language Delta Modification: Given an original code-image pair ($c_1, p_1$) and a natural language instruction ($\delta_{nl}$) describing the change, the model must generate the modified code ($c_2$).

Crucially, the evaluation is conversational: the output of Turn $t$ becomes the input for Turn $t+1$. This tests the model's ability to handle error accumulation and state persistence.

B. Dataset Construction

• Scale: The dataset contains 5,000 difficulty-controlled modification chains (4,142 unique instances with 6 turns each, totaling ~20,710 modification samples) and a 430-example human-verified subset.
• Diversity: It covers 37 distinct chart types (from basic plots to 3D/polar coordinates) and 35 modification types categorized into:
  • Style Modifications (17 types): Colors, titles, markers, themes.
  • Data Modifications (18 types): Filtering, axis scaling, aggregation, rolling averages, chart type conversion.
• Generation Pipeline: An automated pipeline uses LLMs to generate initial charts, then iteratively generates modification instructions and target code/images. All code is executed in a sandboxed Python environment to ensure validity.
• Verification: A hybrid verification system uses Chart-R1 (a fine-tuned VLM) for automated quality assessment and human annotators for the subset to ensure ground truth accuracy.

C. Evaluation Framework

To overcome the limitations of generic LLM judges, the authors propose a composite, grounded evaluation framework:

1. Execution Rate: Measures the percentage of code that executes without errors and produces a valid image.
2. Instruction Following Score:
   • Programmatic: Pass rate of structured assertions (e.g., checking if marker='*' exists in the code).
   • LLM-Judged: Semantic scoring (1–5 scale) for subjective instructions (e.g., "make it professional").
3. Code Quality Score: Pass rate of structural assertions (e.g., presence of imports, savefig, close).
4. Visual Similarity Score: An LLM explicitly enumerates visual differences between the generated and target charts, deducting points for major (e.g., wrong chart type) and minor (e.g., font size) discrepancies.
5. Overall Score: A weighted composite of the above metrics.

3. Key Contributions

1. Formalization of Incremental Editing: The first benchmark to formalize chart editing as a multi-turn task where models update existing code-image pairs while maintaining state, rather than generating from scratch.
2. ChartEditBench Dataset: A large-scale, difficulty-controlled synthetic dataset with 5,000 instances and a human-verified subset, covering a wide spectrum of chart types and modification complexities.
3. Robust Evaluation Metrics: A novel framework combining execution-based fidelity, structured code assertions, and grounded visual similarity analysis, moving beyond unreliable embedding-based or pure LLM-judge metrics.

4. Experimental Results

The authors evaluated six state-of-the-art models (including GPT-5-mini, Claude Haiku 4.5, Qwen3-VL variants, and InternVL3) on the benchmark.

A. Overall Performance

• Top Performers: Claude Haiku 4.5 achieved the highest overall score (2.187), excelling in visual similarity. GPT-5-mini led in instruction following but lagged in visual fidelity.
• Open-Source Efficiency: Qwen3-VL-30B-A3B (using a Mixture-of-Experts architecture with only 3B active parameters) achieved competitive performance (2.139), outperforming larger dense models in some metrics.
• Small Model Limitations: Smaller models (e.g., InternVL3-1B) showed significant gaps, particularly in instruction following (0.480), though they sometimes preserved chart structure even when failing specific edits.

B. Key Findings

1. Error Accumulation: All models exhibited monotonic performance degradation across conversation turns (20–33% drop from Turn 1 to Turn 5). Smaller models degraded more severely, indicating that early errors compound and prevent recovery.
2. Data vs. Style: Models performed significantly better on style modifications (e.g., changing colors) than data-centric transformations (e.g., rolling averages, data filtering). Even top models struggled with rolling averages, often making off-by-one errors or mishandling data shapes.
3. Programmatic vs. Semantic: There is a divergence in capabilities: models like GPT-5-mini excel at precise, programmatic instructions, while others show more stability in semantic (LLM-judged) understanding.
4. Rendering Success: Proprietary models achieved 85–90% rendering success, while open-source models varied widely (46–90%), with complex data operations causing frequent execution failures.

5. Significance

• Real-World Relevance: ChartEditBench shifts the focus from "can the model draw a chart?" to "can the model act as a collaborative analyst?" This mirrors the actual workflow of data professionals who iteratively refine visualizations.
• Diagnostic Power: The benchmark reveals that current MLLMs, while proficient at single-turn generation, lack the robustness required for iterative, stateful programming tasks. The specific failure modes (error accumulation, data reasoning gaps) provide clear directions for future model improvement.
• Evaluation Standard: The proposed hybrid evaluation framework sets a new standard for assessing multimodal code generation, emphasizing execution validity and grounded visual verification over superficial similarity scores.
• Future Directions: The results highlight that improving numerical reasoning and state management in MLLMs is critical for advancing multimodal agents in software engineering and data analysis.