VisCoder2: Building Multi-Language Visualization Coding Agents

1. Problem Statement

While Large Language Models (LLMs) have shown promise in generating code, existing visualization coding agents face significant limitations in practical workflows:

• Limited Language Coverage: Most existing datasets and benchmarks focus heavily on Python (e.g., Matplotlib, Plotly) or declarative specifications like Vega-Lite, neglecting domain-specific languages (DSLs) such as LilyPond (music), Asymptote (vector graphics), LaTeX (typesetting), and Mermaid (diagrams).
• Lack of Executability: Many existing datasets contain code snippets that cannot be reliably executed or rendered, making it difficult to train models on "ground truth" visual outputs.
• Absence of Iterative Correction: Current benchmarks often evaluate single-round generation. However, real-world visualization development is inherently iterative, requiring agents to debug, re-run, and refine code based on execution feedback (runtime errors, rendering failures).
• Benchmark Gaps: Existing benchmarks lack systematic evaluation across multiple languages and do not support multi-round self-debug protocols.

2. Methodology

The authors introduce a comprehensive framework consisting of three core components: a large-scale dataset, a multi-language benchmark, and a family of specialized models.

A. VisCode-Multi-679K (Dataset)

A supervised instruction-tuning dataset comprising 679,000 validated, executable visualization samples and multi-turn correction dialogues across 12 programming languages.

• Data Sources: Aggregated from The-Stack-v2 (general code), svg-diagrams (SVG specific), and CoSyn-400K (synthetic visualization code).
• Pipeline:
  1. Extraction & Filtering: Code blocks are extracted and filtered based on library usage.
  2. Runtime Validation: Code is executed in isolated Jupyter environments (using nbconvert with strict error handling). Only samples that execute successfully and produce valid, non-monochrome images (>10KB) are retained.
  3. Instruction Generation: Natural language instructions are generated using GPT-4.1, structured into five parts: Setup, Data/Visual Description, Data Block, Output Description, and Style Description.
  4. Multi-Turn Integration: Includes 66K multi-turn dialogues from the Code-Feedback dataset to teach models how to revise code based on execution errors.
• Languages Covered: Python, LaTeX, HTML, LilyPond, Asymptote, Vega-Lite, SVG, Mermaid, JavaScript, TypeScript, C++, and R.

B. VisPlotBench (Benchmark)

A systematic evaluation benchmark featuring 888 executable tasks across 8 languages and 13 visual categories (e.g., Bars, Lines, 3D, Music, Hierarchies, Networks).

• Structure: Each task includes a natural language instruction, executable code, and a rendered ground-truth image.
• Evaluation Protocol:
  • Execution Pass Rate: Checks if code runs without error and produces a valid image.
  • Task Score: LLM-based judge evaluates adherence to the natural language instruction.
  • Visual Score: LLM-based judge evaluates perceptual similarity between the generated and ground-truth images.
  • Self-Debug Loop: Models are allowed up to 3 rounds of iterative correction based on execution logs and error messages.

C. VisCoder2 (Model Family)

A family of multi-language visualization models built by fine-tuning Qwen2.5-Coder backbones (3B, 7B, 14B, and 32B parameters) on the VisCode-Multi-679K dataset. The training emphasizes both initial code generation and the ability to perform iterative self-correction.

3. Key Contributions

1. VisCode-Multi-679K: The first large-scale dataset providing executable visualization code across 12 languages, bridging the gap between general coding and domain-specific visualization tasks.
2. VisPlotBench: A standardized benchmark enabling the evaluation of both initial generation and multi-round self-debug capabilities across diverse visualization languages and categories.
3. VisCoder2 Models: A new family of open-source models that significantly outperform size-matched baselines and approach the performance of proprietary models (like GPT-4.1) in executable visualization tasks.
4. Insight on Iterative Refinement: Demonstrated that self-debug mechanisms are critical for handling symbolic and compiler-dependent languages, often yielding larger performance gains than scaling model size alone.

4. Key Results

Experiments were conducted on VisPlotBench comparing VisCoder2 against proprietary models (GPT-4.1, GPT-4.1-mini) and open-source baselines (DeepSeek-Coder, Qwen2.5-Coder).

• Overall Performance:
  • VisCoder2-32B achieved an 82.4% execution pass rate with self-debug, matching GPT-4.1 (82.4%) and surpassing GPT-4.1-mini (81.1%).
  • Without self-debug, VisCoder2-32B scored 73.1%, significantly outperforming the base Qwen2.5-Coder-32B (57.5%).
• Language-Specific Gains:
  • Symbolic Languages: VisCoder2 showed massive improvements in languages like LilyPond (69.1% vs. 5.5% for baseline) and LaTeX (42.9% vs. 29.5% for baseline), where previous models struggled due to strict syntax requirements.
  • Declarative Languages: High performance on Vega-Lite (94.6%) and HTML (91.7%).
• Impact of Self-Debug:
  • Self-debug consistently improved execution pass rates by ~10 percentage points across all model scales.
  • It was most effective for fixing structural and interface errors (e.g., missing tokens, invalid arguments) in symbolic languages.
• Ablation Studies:
  • Training on CoSyn-400K (synthetic data) was crucial for symbolic languages (LilyPond, Asymptote).
  • Training on Code-Feedback (multi-turn dialogues) was essential for improving self-debug capabilities.
  • Combining all data sources yielded the best results, confirming the need for a diverse mix of natural, synthetic, and interactive data.

5. Significance and Conclusion

• Bridging the Gap: VisCoder2 narrows the performance gap between open-source and proprietary models for complex, multi-language visualization tasks, making high-quality visualization agents more accessible.
• Iterative Workflows: The paper establishes that reliable visualization coding requires not just generation but robust iterative refinement. The "generate-execute-repair" loop is shown to be a critical mechanism for handling the complexity of symbolic and compiler-dependent languages.
• Future Directions: The work highlights that while progress has been made, symbolic languages remain challenging due to deep semantic inconsistencies and compiler-specific errors that current self-debug protocols struggle to resolve. The authors call for grammar-aware training objectives and more robust runtime integrations.

In summary, VisCoder2 represents a significant step forward in building reliable, multi-language coding agents for data visualization, providing the necessary data, benchmarks, and models to support real-world, iterative development workflows.