ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats

This paper introduces ChartArena, a comprehensive bilingual benchmark featuring eight chart families across digital, printed, and hand-drawn scenarios with a format-agnostic evaluation protocol, to systematically assess and reveal the current capabilities and limitations of 26 leading multimodal large language models in parsing diverse chart types.

The Gist

Imagine you have a giant library of charts, graphs, and diagrams. Some are neat computer drawings, some are photos of papers taken in a messy office, and some are rough sketches drawn on a whiteboard. Now, imagine you want to teach a robot to read these pictures and turn them into a list of facts (like a spreadsheet) or a map of connections (like a family tree).

This paper introduces ChartArena, a massive new "test track" designed to see how good different robots (AI models) are at this task.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Language Barrier" and the "Clean Room" Issue

Before this paper, testing these robots was like trying to compare runners in a race where:

• The Rules Changed: One runner had to write their answer in English, another in Spanish, and a third in Morse code. You couldn't easily compare who was faster because the answers looked so different.
• The Track Was Fake: Most tests only used perfect, computer-generated charts. It was like training a driver only on a smooth, empty racetrack, then expecting them to drive perfectly in the rain on a bumpy dirt road. Real life has blurry photos, crooked angles, and messy handwriting, but the old tests ignored this.
• The Scope Was Narrow: The tests mostly looked at simple bar graphs and pie charts. They ignored complex diagrams like flowcharts (decision trees) or mind maps, which are like tangled webs of ideas rather than simple numbers.

2. The Solution: ChartArena (The Ultimate Obstacle Course)

The authors built ChartArena, a new, super-comprehensive test that fixes all the above problems.

• Eight Different "Obstacles": The test covers eight types of charts, from simple number charts (bar, line, pie) to complex structural diagrams (flowcharts, mind maps).
• Three "Weather Conditions": Every chart is tested in three ways:
  1. Digital: A perfect, crisp computer image.
  2. Printed: A photo of a paper document (which might be slightly blurry or tilted).
  3. Hand-Drawn: A photo of a sketch on a whiteboard or notebook (messy ink, uneven lines).
• Two Languages: The test is bilingual, covering both English and Chinese.
• The "Human-Agent" Team: To make sure the answers are correct, they used a team approach. An AI made a first draft of the answer, and then human experts checked and fixed it multiple times. This ensures the "gold standard" answers are reliable.

3. The Scoring System: The "Universal Translator"

Since different robots output answers in different formats (some write code, some write tables, some write lists), how do you score them fairly?

The authors created a Universal Translator.

• For Number Charts: No matter if the robot wrote a Python script, a CSV file, or a Markdown table, the system translates it all into a simple list of "Who, What, How Much" (Triples).
• For Diagrams: No matter if the robot used Mermaid, Graphviz, or PlantUML, the system translates it all into a map of dots and lines (a Directed Graph).

Once everything is translated into this common language, the system scores them. It doesn't just check if the words match exactly; it checks if the structure makes sense. It's like grading a student's essay: if they use the right synonyms and get the main idea right, they get points, even if the spelling isn't perfect.

4. The Results: Who Won the Race?

The authors tested 26 different AI models on this new track. Here is what they found:

• The "Big Tech" Robots are Leading: The most advanced, paid models (like Gemini 3.1 Pro) are currently the best at the job. However, the best free, open-source models are catching up very fast.
• The "Document Readers" are One-Trick Ponies: Some models are great at reading documents and simple number charts. But when you show them a complex flowchart or a mind map, they get lost. They lack the "world knowledge" to understand how ideas connect.
• The "Specialists" are Too Specialized: There are models built specifically for charts. While they are okay at simple bar graphs, they often fail completely when faced with diagrams or hand-drawn sketches. They haven't learned enough variety to handle the real world.
• The Hardest Challenges:
  • Radar Charts: These circular charts (like a spider web) are the hardest for everyone to read.
  • Hand-Drawn Sketches: When the input is a messy photo of a sketch, performance drops significantly for all models.

5. The Takeaway

The paper concludes that while AI is getting better at reading charts, there is still a big gap between what they can do in a perfect lab setting and what they can do in the messy real world.

ChartArena provides a fair, unified way to measure progress. It shows us exactly where the robots are failing (complex diagrams, messy photos) so developers know where to focus their efforts to build truly reliable chart-reading AI.

In short: We finally have a fair race track with real-world obstacles, and we now know exactly which robots are ready for the real world and which ones still need more training.

Technical Summary

Technical Summary: ChartArena

Problem Statement

Despite the rapid advancement of Multimodal Large Language Models (MLLMs) in chart parsing, the field lacks a unified, comprehensive evaluation framework. Current research is hindered by three compounding fragmentation issues:

1. Output Incompatibility: Models generate results in mutually incompatible syntactic formats (e.g., Markdown, JSON, CSV, Python code, SVG, Mermaid), rendering direct cross-model comparison intractable.
2. Narrow Benchmark Scope: Existing benchmarks focus predominantly on a limited set of numeric charts (bar, line, pie) and largely ignore structurally distinct diagrammatic charts (flowcharts, mind maps) that require graph-level understanding.
3. Idealized Data Conditions: Datasets are dominated by pristine digital renderings, failing to account for real-world visual perturbations such as printed photos (with lighting/perspective distortion) and hand-drawn sketches (with ink inconsistencies and noise).

These gaps prevent the community from obtaining an honest assessment of model capabilities across diverse chart types, languages, and visual scenarios.

Methodology

To address these challenges, the authors introduce ChartArena, a comprehensive bilingual benchmark, and a format-agnostic evaluation protocol.

1. ChartArena Benchmark

• Coverage: The benchmark spans eight chart families: six numeric types (bar, line, pie, radar, box plot, combination chart) and two diagrammatic types (flowchart, mind map).
• Visual Scenarios: Each chart type is evaluated across three distinct visual conditions:
  • Digital renderings (clean).
  • Printed photos (captured from physical documents).
  • Hand-drawn photos (sketched on whiteboards/notebooks).
• Languages: All content is provided in both English and Chinese.
• Dataset Construction: The dataset comprises 2,400 samples (50 per setting). It is constructed via a human-agent collaborative annotation pipeline:
  • MLLMs generate coarse drafts (Markdown for numeric, Mermaid for diagrams).
  • Human annotators perform multi-stage verification to correct structural elements (nodes, edges, axes) and semantic content (labels, values).
  • Disagreements on ambiguous values are reconciled by multiple annotators to ensure high reliability.

2. Format-Agnostic Evaluation Protocol

To enable fair comparison across models with heterogeneous outputs, the protocol normalizes all predictions into two canonical semantic spaces before scoring:

• Normalized Triple View (Numeric Charts): All outputs (Markdown, CSV, JSON, Code, etc.) are mapped to a set of semantic triples (header, entity, value). This step includes lexical and numeric canonicalization (e.g., treating "3.0" and "3" as identical).
• Directed Graph View (Diagrammatic Charts): Diagram languages (Mermaid, Graphviz, PlantUML, etc.) are normalized into a directed graph with labeled nodes and edges, abstracting away syntactic differences to capture topological and hierarchical structures.

Scoring Metrics:

• Structure-Aware Metrics: Scoring is based on structural correctness rather than string similarity.
  • Numeric: Intersection-over-Union (IoU) of triple sets, using Levenshtein distance for text and relative-error thresholds for numbers.
  • Diagrammatic: Node and edge matching via the Hungarian algorithm, with higher weight on edges to penalize topological errors.
• Aggregation: Results are reported as Exact Match (EM) and mean Average Precision (mAP) across three tolerance levels (strict, slight, high). The primary metric reported is mAPhigh.

Key Results

The authors evaluated 26 leading models, categorized into General-purpose MLLMs, Document Parsing MLLMs, and Expert Chart Parsers.

1. Proprietary vs. Open-Source: Frontier proprietary models (e.g., Gemini 3.1 Pro) lead overall (59.2 EN / 73.2 ZH average mAP). However, the strongest open-source systems (e.g., Qwen3.5-35B-A3B, Kimi K2.5) are rapidly closing the gap, achieving competitive scores in the top tier.
2. Numeric vs. Diagrammatic Performance:
   • Document Parsing MLLMs (e.g., HunyuanOCR) handle numeric charts reasonably well but fall sharply behind on diagrammatic structures (flowcharts, mind maps). This suggests that inferring implicit nodes and hierarchies requires broader world knowledge found in larger general-purpose models.
   • Expert Chart Parsers remain limited to narrow chart families. Many dedicated parsers (e.g., ChartAst, TinyChart) cannot handle diagrammatic charts at all, and even those that can (e.g., RRVF) lag significantly behind general-purpose MLLMs in overall performance.
3. Specific Challenges:
   • Radar Charts: Identified as the most difficult numeric category across all models, likely due to the difficulty of estimating angular values and scarcity in training data.
   • Hand-Drawn Scenarios: Performance degrades substantially for hand-drawn inputs compared to digital renderings.
   • Format Stability: The evaluation protocol proves robust for most formats (Markdown, JSON, Mermaid), but SVG (for numeric charts) and PlantUML (for flowcharts) show significant performance drops, indicating specific format-structure compatibility issues.

Significance and Claims

The paper claims that ChartArena provides the first unified foundation for evaluating chart parsing across the full spectrum of chart types, visual scenarios, and languages. Its significance lies in:

• Exposing Capability Gaps: It reveals that current "expert" parsers are not yet general-purpose and that even leading models struggle with diagrammatic structures and non-ideal visual conditions.
• Standardizing Evaluation: By introducing a deterministic, format-agnostic normalization protocol, it enables fair cross-paradigm comparison, moving the field away from fragmented, format-specific benchmarks.
• Realistic Assessment: By including printed and hand-drawn scenarios, it shifts the evaluation focus from controlled laboratory settings to realistic deployment conditions.

The authors position ChartArena as a "lasting testbed" intended to guide future research toward more reliable, reproducible, and truly general-purpose chart understanding systems.