ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding

1. Problem Statement

Current Vision-Language Models (VLMs) struggle with chart understanding because interpreting visualizations requires the simultaneous integration of three distinct modalities:

1. Geometric visual patterns (shapes, colors, layouts).
2. Structured numerical data (underlying values, trends).
3. Natural language (titles, labels, reasoning).

Existing datasets suffer from critical limitations:

• Scale and Diversity: Most are small (thousands rather than millions) and cover only a few chart types (e.g., bar, line, pie).
• Modality Gaps: Many lack critical components like executable plotting code, grounding annotations (bounding boxes), or reasoning traces.
• Task Narrowness: Datasets often focus on a single task (e.g., QA or captioning) rather than the full spectrum of chart comprehension.
• Performance Ceiling: Open-source models lag behind proprietary systems (like GPT-4o) in complex reasoning tasks due to a lack of high-quality, code-aligned supervision.

2. Methodology: The ChartNet Pipeline

The authors introduce ChartNet, a dataset of 1.5 million high-quality multimodal tuples generated via a novel code-guided synthesis pipeline. The pipeline operates in five stages:

1. Seed Selection & Chart-to-Code Reconstruction:
   • Starting with 150,000 seed chart images (from TinyChart), a VLM (Pixtral-Large) is prompted to generate executable Python plotting code that reconstructs the image.
2. Code-Guided Augmentation:
   • An LLM (GPT-OSS-120b) iteratively rewrites the generated code to create diverse variations. It changes the chart type (from 24 available types), plotting library (6 libraries), data values, labels, and styles while maintaining semantic relevance. This is the primary driver of dataset scaling.
3. Chart Rendering:
   • The augmented code is executed to render new chart images. Only successfully executed scripts are paired with their rendered images.
4. Quality Filtering:
   • A VLM evaluates rendered images for defects (overlapping text, cropped labels, obscured features). Images with visual errors and their corresponding code are discarded. This reduced visual error rates from ~15% to ~6%.
5. Attribute Generation:
   • Using the code as a "ground truth" context, the system extracts tabular data (CSV), generates natural language summaries, and creates Question-Answer (QA) pairs with Chain-of-Thought (CoT) reasoning.

Dataset Composition:

• Core Synthetic Data: 1.5M tuples containing: Chart Image, Plotting Code, Data Table (CSV), Text Summary, and QA with CoT Reasoning.
• Specialized Subsets:
  • Human-Annotated: ~96k samples verified by domain experts.
  • Real-World: 30k charts from reputable sources (World Bank, Pew Research, etc.) covering economics, geopolitics, and science.
  • Grounding: QA pairs with bounding box annotations for spatial reasoning.
  • Safety: 7,000 training samples with adversarial prompts to mitigate harmful/biased reasoning.

3. Key Contributions

1. Code-Guided Generation Pipeline: A novel approach that treats executable code as the intermediate representation for data visualization, enabling scalable, diverse, and semantically consistent data generation.
2. ChartNet Dataset: The largest open-source chart dataset to date, featuring:
   • Scale: 1.5 million samples.
   • Diversity: 24 chart types and 6 plotting libraries.
   • Completeness: Full alignment of image, code, data, text, and reasoning.
3. Empirical Validation: Comprehensive experiments demonstrating that fine-tuning on ChartNet significantly boosts performance across various model sizes and tasks.

4. Results

The authors evaluated models ranging from 256M to 7B parameters, comparing them against off-the-shelf open-source models and proprietary systems like GPT-4o.

• Performance Gains: Fine-tuning on ChartNet yielded consistent, substantial improvements across all tasks:
  • Chart Reconstruction (Code Gen): Execution rates and data fidelity improved dramatically. For example, the 2B Granite-Vision model reached >90% on most metrics, while the 7B LLaVA model saw a +42.4 point gain in code data fidelity.
  • Data Extraction: Models improved significantly in recovering numeric tables from images. The fine-tuned 7B model (70.3%) outperformed GPT-4o (46.7%) in this specific task.
  • Summarization & QA: Fine-tuned models surpassed GPT-4o and larger open-source baselines (e.g., 72B models) in summarization and complex CoT reasoning.
• Generalization: Models trained on synthetic ChartNet data showed strong transferability to real-world public benchmarks (ChartCap and ChartMimic-v2), proving the synthetic data's utility for real-world scenarios.
• Efficiency: A 2B or 7B model fine-tuned on ChartNet consistently outperformed 20B–72B models trained on conventional multimodal corpora, suggesting that high-quality, code-aligned supervision is more effective than simply scaling model parameters for chart understanding.

5. Significance

• New Standard for Chart Understanding: ChartNet unifies scale, diversity, and representational completeness, addressing the "bottleneck" of high-fidelity supervision in this domain.
• Shift in Paradigm: The results suggest that for tightly coupled visual-numerical-linguistic tasks, the quality and alignment of training data (specifically via code) are more critical than model size.
• Foundation for Future Models: By providing a scalable, open foundation, ChartNet enables the development of models capable of robust, interpretable, and numerically grounded reasoning over data visualizations, moving beyond simple description to deep understanding.
• Safety and Grounding: The inclusion of safety and grounding subsets addresses critical gaps in current VLMs regarding bias, hallucination, and spatial reasoning.

The dataset is publicly available at Hugging Face (ibm-granite/ChartNet).