Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction

1. Problem Statement

The paper addresses the challenge of Chart-to-Code Generation, a task where Multimodal Large Language Models (MLLMs) must generate executable code (e.g., Python/Matplotlib) to reproduce a given input chart image.

• Current Limitations: While MLLMs excel at general visual understanding, they struggle with chart-to-code tasks due to the high information density, complex layouts, and the need for precise translation of visual elements into structured code.
• Existing Approaches:
  • Direct Prompting: Often yields suboptimal results due to the complexity of the task.
  • ChartCoder: Uses step-by-step decomposition but requires training, making it incompatible with closed-source models (e.g., GPT-4o).
  • METAL: A training-free, iterative multi-agent framework. However, it suffers from unbalanced optimization because it updates code based on a single, lowest-performing metric per iteration. This can lead to the degradation of other visual aspects (e.g., optimizing color might ruin layout).

2. Methodology: ChartIR

The authors propose ChartIR, a training-free, model-agnostic framework that improves chart-to-code generation through a two-stage iterative process driven by structured instructions.

Core Philosophy

The method decomposes the problem into two distinct capabilities:

1. Visual Understanding: Extracting multi-dimensional information (structure, text, color, layout).
2. Code Translation: Converting this information into executable code.

Stage 1: Initial Code Generation

Instead of prompting the model directly with the image, ChartIR introduces a Structured Chart Description phase:

• Description Generation: An MLLM (MLLM-Describer) analyzes the reference chart ($I_r$) to generate a textual description ($\delta$).
  • For closed-source models (e.g., GPT-4o), the model is prompted directly to generate the description.
  • For open-source models (e.g., Qwen2-VL), a specialized fine-tuned version of Qwen2.5-VL is used. This model was trained on ~5,000 (chart, description) pairs derived from the ChartX dataset to ensure high-quality semantic extraction.
• Initial Code: The MLLM-Coder receives both the reference image ($I_r$) and the structured description ($\delta$) to generate the initial code ($C_g^0$).

Stage 2: Iterative Refinement

This stage progressively improves the code by identifying and correcting discrepancies between the generated chart and the reference.

• Difference Generation: The model compares the currently rendered chart ($I_g^t$) with the reference ($I_r$) and generates a Difference Description ($\mu$). This explicitly highlights discrepancies in text, color, axes, and layout.
• Code Update: The model generates new code ($C_g^{t+1}$) using the reference image, the initial description, the current code, and the difference description.
• Update Criterion: The new code is accepted only if the visual discrepancy score between the new chart and the reference is lower than the previous one. The discrepancy is calculated via a weighted average of multiple metrics (CLIP, DINO, SSIM, PSNR, Hamming Distance) to ensure holistic improvement, avoiding the single-metric trap of METAL.
• Convergence: The process stops when the discrepancy score fails to improve for a set number of iterations (convergence counter).
• Debugging: A specialized code-fixing LLM is invoked to handle execution errors automatically during the loop.

3. Key Contributions

1. ChartIR Framework: A novel, training-free, model-agnostic framework that significantly enhances chart-to-code generation for both open-source and closed-source MLLMs.
2. Structured Instruction Mechanism: The introduction of two specific instruction types:
   • Description: Captures global visual elements.
   • Difference: Characterizes specific discrepancies between the generated and reference charts.
   • These transform visual features into language representations, bridging the gap between vision and code.
3. Holistic Iterative Refinement: Unlike previous methods that optimize single metrics, ChartIR uses a multi-metric discrepancy evaluation to ensure balanced optimization across all visual dimensions (text, layout, color, type).

4. Experimental Results

The method was evaluated on two benchmarks: Plot2Code and ChartMimic, using Qwen2-VL (open-source) and GPT-4o (closed-source).

• Performance Gains: ChartIR consistently outperformed both "Direct Generation" and "METAL" across almost all metrics.
  • GPT-4o Score: On Plot2Code, ChartIR achieved 6.56 vs. METAL's 6.02 and Direct's 5.61. On ChartMimic, it reached 7.15 vs. 6.89 (METAL).
  • Low-Level Metrics: Significant improvements were observed in Text, Layout, Type, and Color scores.
  • Image Similarity: ChartIR showed superior results in traditional metrics like PSNR, SSIM, and MSE, indicating better visual fidelity.
• Ablation Studies:
  • Removing the Description component caused a significant drop in performance (e.g., GPT-4o score dropped from 3.96 to 3.10 for Qwen2-VL), proving the necessity of semantic context.
  • Removing the Difference component also degraded performance, confirming that explicit contrastive feedback is crucial for iterative refinement.
  • The combination of both components yielded the best results, validating the synergistic effect of the framework.

5. Significance and Conclusion

• Generalizability: ChartIR is unique in its ability to boost performance on closed-source models (where training is impossible) while also improving open-source models via fine-tuning the description module.
• Quality of Output: The method produces code that is not only structurally accurate but also visually faithful, aligning closely with human judgment (as measured by GPT-4o scores).
• Future Impact: By decoupling visual understanding from code generation and using structured intermediate representations, ChartIR provides a robust blueprint for future multimodal reasoning systems, particularly for tasks requiring high-precision visual-to-synthetic translation.

Limitations Noted: The method incurs higher computational costs due to multiple inference steps (though fewer queries on closed models compared to some baselines) and shows slightly diminishing returns on already powerful closed-source models compared to open-source ones, suggesting room for further instruction optimization.