Technical Summary: PAINTBENCH

Problem Statement

While current multimodal models demonstrate proficiency in open-ended, subjective visual editing (e.g., "make the sky more blue"), they struggle with precise, single-answer visual editing tasks. Existing benchmarks for image editing largely rely on human judgment, vision-language model (VLM) scoring, or perceptual metrics (e.g., SSIM, LPIPS, FID). These approaches are ill-suited for tasks with deterministic ground truths, as they introduce bias and fail to verify exact correctness. The paper identifies a gap in evaluating whether models can execute exact raster operations—such as moving a shape 50 pixels, flood-filling a specific region, or removing a specific instance—where a unique correct output exists.

Methodology

PAINTBENCH: A Deterministic Benchmark

The authors introduce PAINTBENCH, a procedurally generated benchmark designed to evaluate 20 fundamental precise visual editing operations across four categories:

1. Geometric Transformation: Translation, rotation, reflection, scaling, shearing.
2. Structural Manipulation: Construction, removal, copying, border, cropping.
3. Color Change: Recolor, flood fill, blending, gradient, point operations.
4. Symbolic Reasoning: Comparison, ordering, pattern, counting, legend.

Generation and Design:

• Procedural Generation: Problems are generated from random seeds with configurable parameters (canvas dimensions, object count, background texture, color palette). This creates an effectively infinite, contamination-resistant evaluation suite.
• Deterministic Ground Truth: Each problem consists of an input image, a natural language instruction, and a unique ground-truth answer image generated by a deterministic transformation $A = f(I, t)$.
• Visual Conditions: To test robustness, every task is evaluated under eight visual conditions, varying one parameter at a time (e.g., striped vs. solid backgrounds, object counts from 3 to 60, nonstandard color palettes).

Evaluation Protocol:

• Pixel-Level Comparison: The evaluation avoids "judge models" entirely. Instead, it compares the model's output $\hat{A}$ against the ground truth $A$ and input $I$ at the pixel level.
• Color Distance: Per-pixel correctness is measured using $\Delta E^*_{76}$ (Euclidean distance in CIE Lab* space). A pixel is considered correct if the distance is within a tolerance $t \in \{0, \dots, 10\}$.
• Metrics:
  • IoU@t: The Jaccard index applied to pixel correctness, jointly penalizing failure to edit (incorrectly edited pixels) and corruption of the preservation region (incorrectly preserved pixels).
  • mIoU (Mean IoU): The primary summary metric, averaging IoU@t across 11 color tolerances ($t=0$ to $10$) and all problems. This captures performance across the full spectrum of color precision.

TINYGRAFIXBENCH

To test generalization beyond synthetic shapes, the authors introduce TINYGRAFIXBENCH, a companion benchmark for data visualization editing. It applies the same procedural, deterministic framework to five chart types (bar, scatter, line, heatmap, network) with 600 problems.

Key Results

Overall Performance

Across 11 evaluated models (spanning native multimodal generators, diffusion editors, and flow-matching editors), performance is extremely low:

• The best-performing model (NANO-BANANA-2) achieves only 17.1% mIoU.
• The second-best (GPT-IMAGE-2) scores 16.3%.
• Open-weight models generally score lower, with some (e.g., HUNYUANIMAGE-3.0, INSTRUCTPIX2PIX) scoring near 0% on many tasks.

Task Difficulty and Specialization

• Hardest Tasks: Geometric transformations (especially shearing and scaling), formula-based color changes (gradients, blending), and most structural manipulations are consistently difficult for all models.
• Easier Tasks: Removal and single-color operations (recolor, flood fill) are more tractable but still far from solved (e.g., Removal reaches ~50% for the best model).
• Model Specialization: Models exhibit distinct strengths. For instance, GPT-IMAGE-2 excels in geometric and structural tasks, while NANO-BANANA-2 leads in color change and symbolic reasoning. Notably, NANO-BANANA-2 scores 47.1% on "Legend" tasks compared to 19.4% for GPT-IMAGE-2.

Sensitivity to Scene Variation

Models are highly brittle to scene variations:

• Striped Backgrounds: Cause significant mIoU drops (e.g., -11.1% for NANO-BANANA-2).
• High Object Counts: Performance degrades linearly as object count increases (e.g., -11.9% drop for NANO-BANANA-2 when moving from 3 to 60 objects).
• Nonstandard Colors: While headline mIoU scores appear stable, exact pixel-match accuracy ($\Delta E^*_{76} = 0$) collapses for nonstandard palettes, indicating models struggle to reproduce specific, non-standard colors precisely.
• Edit-Region Size: Models significantly over-edit relative to the target region. For small edit regions (< 32² pixels), models change 50–1,400 times more pixels than required, driving sharp performance declines.

Generalization

There is a strong linear correlation ($R^2 = 0.91, p < 0.001$) between PAINTBENCH and TINYGRAFIXBENCH scores. This suggests that PAINTBENCH's synthetic benchmarks capture fundamental capabilities that generalize to applied data visualization editing tasks.

Significance and Claims

The paper claims that PAINTBENCH provides a rigorous, bias-free foundation for measuring progress in precise multimodal visual editing. By shifting from subjective, open-ended evaluation to deterministic, pixel-level verification, the benchmark reveals that current state-of-the-art models are fundamentally incapable of executing basic raster operations with high precision.

The authors emphasize that:

1. Current models are not "solved" for precise editing: Even the best models fail the majority of tasks, particularly those requiring geometric precision or complex color logic.
2. Evaluation must be task-specific: Aggregate rankings hide significant model specializations and failure modes (e.g., color imprecision vs. structural omission).
3. Procedural generation is essential: Static benchmarks are prone to contamination; dynamic generation allows for controlled ablation studies of scene variations (backgrounds, object counts, colors).
4. The framework is extensible: The approach used for PAINTBENCH and TINYGRAFIXBENCH can be applied to other domains with unique correct outputs, such as scientific visualization, engineering drawing, and 3D scene manipulation.

The paper concludes that while open-ended generation has advanced, the ability to perform exact, deterministic visual edits remains a critical, unsolved challenge in multimodal AI.