💬 NLP

VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

This paper introduces VisEditBench, a comprehensive benchmark of 1,395 human-annotated tasks designed to evaluate Vision-Language Models' ability to edit visualization code based on multimodal feedback, revealing significant performance gaps in current models and proposing VisEditAgent, a render-grounded framework that substantially improves editing accuracy.

Original authors: Mizanur Rahman, Arshia Azimlu, Shadikur Rahman, Md Tahmid Rahman Laskar, Amran Bhuiyan, Shafiq Joty, Enamul Hoque Prince

Published 2026-08-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Mizanur Rahman, Arshia Azimlu, Shadikur Rahman, Md Tahmid Rahman Laskar, Amran Bhuiyan, Shafiq Joty, Enamul Hoque Prince

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has just learned to cook a perfect omelet from a recipe book. You can follow the instructions, crack the eggs, and flip the pan. But what happens when your friend takes a bite, grimaces, and says, "It's too salty, and the cheese is burnt on the bottom"? Or what if they hand you a photo of a different omelet and say, "I want mine to look exactly like this one"? Suddenly, your job isn't just about following a recipe; it's about fixing a mistake or copying a style while keeping the eggs and the heat just right. This is the world of data visualization, where computers try to turn numbers into pictures like charts and graphs.

For a while, scientists have taught computers how to write the "recipes" (code) to make these pictures from scratch. But in the real world, we rarely just want a brand-new picture; we usually have an existing one that looks a little weird, or we want to change its style to match a report. This is called multimodal feedback: using a mix of text instructions, the actual picture of the chart, and the code that made it to tell the computer what to fix. The big question is: Can these smart computers actually "see" what's wrong with a chart and fix it without breaking the data underneath?

This paper, titled VisEditBench, introduces a new playground to test exactly that. The researchers built a massive collection of 1,395 real-world editing challenges, like a "driving test" for AI chart-makers. They found that while the best AI models are getting better at writing code, they are still quite clumsy at fixing charts based on visual feedback. The top model, Claude-4.6-Sonnet, managed to pass about 74.46% of the tests, but most open-source models struggled, staying below 50%. The hardest part? Changing the style of a chart to match a reference image; even the best model only got 55.71% right.

To help these models get better, the authors created a new tool called VisEditAgent. Think of this as giving the AI a "try, fail, and fix" loop. Instead of guessing the answer once, the agent writes a few different versions of the code, actually draws the charts, checks if they look right, and then tweaks them until they are perfect. When they used this method with a strong model, the success rate jumped from 55.75% to 67.99%. This suggests that the secret to fixing charts isn't just being smart; it's being able to look at your work, realize it's wrong, and try again. The paper concludes that while we are making progress, teaching computers to edit visualizations with the same care a human designer would is still a work in progress.

Technical Summary: VisEditBench

Problem Statement

While Vision-Language Models (VLMs) have demonstrated strong capabilities in generating visualization code from scratch (text-to-visualization) or reconstructing code from chart images (chart-to-code), they have largely been evaluated on one-shot generation tasks. Real-world visualization authoring is inherently iterative; users frequently need to revise existing visualizations to repair flaws, improve readability, or adapt styles based on multimodal feedback.

Current benchmarks fail to capture this iterative editing workflow. They do not evaluate the ability of models to take existing visualization code, a rendered chart (often with visual flaws or marked regions), and natural language instructions to produce revised, executable code that preserves data semantics while fixing visual issues. This gap leaves the challenge of "visually grounded code editing" largely unexplored.

Methodology

1. VisEditBench Dataset

The authors introduce VisEditBench, a benchmark comprising 1,395 human-annotated visualization code-editing tasks. The dataset is constructed from two primary sources to ensure realism:

Real-world issues: 120 cases manually collected from Stack Overflow and GitHub issues (Matplotlib/Vega-Lite) involving user-reported bugs.
Model failures: Cases generated by rerunning frontier and open-source models on the Text2Vis benchmark, where the generated outputs exhibited visual, semantic, or presentation errors.

Task Formulation:
Each task $x_i$ is defined as $(c_i, I_i, u_i, c'_i)$ , where:

$c_i$ : Input visualization code.
$I_i$ : Rendered chart image (either a buggy/marked chart or a reference chart).
$u_i$ : Natural language editing instruction.
$c'_i$ : Target revised executable code.

Editing Settings:
The benchmark covers two practical scenarios:

Feedback-guided repair: The model receives a flawed chart and textual feedback to repair the visualization while preserving intent.
Reference-guided restyling: The model receives a target chart image and must adapt the original code to match the reference style without altering data semantics.

Taxonomy:
Tasks are categorized into eight editing intents: Correctness Repair, Quality Improvement, Robustness/Generalization, Style Adaptation, Constraint Satisfaction, Consistency Harmonization, Refactor/Transformation, and Style-aware Error Repair. The dataset spans diverse chart types (Bar, Line, Boxplot, Scatter, etc.) and libraries (Matplotlib and Vega-Lite).

2. Evaluation Framework

The authors propose a structured evaluation measuring:

Code Execution: Whether the code runs and renders a valid chart.
Task Accuracy: Whether the edit satisfies the instruction while preserving semantics.
Readability & Clarity: Legibility of labels, legends, and axes.
Visual Quality: Overall presentation, layout, and polish.
Visual Similarity: Alignment with the target/reference chart.
Final Pass Rate: A strict metric requiring successful execution, high task accuracy (≥4.5), high readability/quality (≥4.0), and high visual similarity (≥90).

An automatic VLM-based evaluator (GPT-4o, or Gemini 2.5 Pro for GPT-4o outputs to avoid bias) is used with a fixed rubric, validated against human judgments.

3. VisEditAgent

To establish a strong baseline and demonstrate the value of iterative refinement, the authors propose VisEditAgent. Unlike zero-shot prompting, this framework mimics the human iterative workflow:

Edit Planning: Identifies visual issues and relevant code regions.
Candidate Generation: Generates multiple code variations.
Execution & Rendering: Executes candidates to produce visual outputs.
Visual Validation: Evaluates candidates against the instruction and reference.
Refinement: Iteratively refines the best candidate based on execution diagnostics and visual feedback.

Key Results

Zero-Shot Performance

The authors evaluated 20 state-of-the-art VLMs (both closed-source and open-source).

Performance Gap: There is a significant gap between generating executable code and performing visually faithful editing. Even the best model, Claude-4.6-Sonnet, achieved an overall pass rate of 74.46% in zero-shot settings, while most open-source models remained below 50%.
Executability vs. Visual Fidelity: Strong models often produce executable code (e.g., Claude-4.6-Sonnet: 96.19% execution success) but fail on visual similarity (89.42%), indicating they struggle to preserve layout fidelity and stylistic alignment.
Task Difficulty: Performance varies sharply by intent. Models perform relatively well on correctness repair (70.34% for Claude-4.6-Sonnet) but struggle significantly with visually grounded style adaptation in zero-shot settings, where even the best model achieves only 55.71%.

VisEditAgent Results

Using GPT-4o as the base model, VisEditAgent improved the overall pass rate from 55.75% (zero-shot) to 67.99%.

Iterative Refinement: The largest gains were observed in visually grounded tasks. Style adaptation improved from 10.00% (GPT-4o zero-shot) to 62.85%, and consistency harmonization from 47.46% to 64.41%.
Ablation: Removing multi-candidate generation or the refinement step significantly degraded performance, confirming that iterative validation against rendered feedback is critical for successful editing.

Significance and Contributions

The paper claims four primary contributions:

VisEditBench: The first benchmark specifically for evaluating visualization code editing from multimodal feedback, covering 1,395 tasks across diverse intents and libraries.
Structured Evaluation: A comprehensive framework measuring executability, task accuracy, readability, visual quality, and visual similarity, moving beyond simple code generation metrics.
Empirical Analysis: Zero-shot evaluations of 20 VLMs revealing that while models can generate code, they lack the multimodal reasoning required for faithful, iterative visualization editing, particularly in style adaptation.
VisEditAgent: A render-grounded baseline demonstrating that iterative generation, execution, and visual validation significantly improve editing performance, suggesting that reliable visualization authoring requires multimodal reasoning loops rather than single-pass generation.

The authors conclude that VisEditBench and VisEditAgent establish a foundation for developing user-aligned, feedback-aware visualization systems, highlighting that the field must move beyond one-shot generation to address the iterative nature of real-world data visualization.