Technical Summary: Chartography: A Benchmark for Professional Chart Understanding

Problem Statement

Professionals in fields such as medicine, engineering, finance, and manufacturing frequently rely on charts to make consequential decisions. However, existing benchmarks for chart understanding (e.g., ChartQA, CharXiv, ChartMuseum) fail to adequately measure the capabilities required for these professional tasks. Current benchmarks are characterized by:

• Format Limitations: A dominance of simple bar, line, and pie charts, lacking domain-specific formats like Kaplan-Meier curves, contour maps, Sankey diagrams, and 3D surface plots.
• Reasoning Depth: Reliance on short reasoning chains and constrained answer formats (e.g., multiple-choice) that reduce the answer space.
• Saturation: Frontier models already achieve 80–90% accuracy on these benchmarks, with many errors attributed to defects in ground truth rather than genuine model failures.
• Grading Issues: Evaluation often relies on printed labels or uniform percentage tolerances rather than calibrated visual estimation appropriate for the specific chart's precision.

Consequently, high scores on existing benchmarks do not correlate with the ability to interpret charts in real-world professional settings where data is not provided in structured tables and domain conventions must be applied.

Methodology

The Chartography Benchmark

Chartography is a benchmark consisting of 100 tasks designed to simulate professional chart reading.

• Task Construction: Each task pairs a chart image with a question written by a working professional in the relevant domain (e.g., oncologists, structural engineers, traders).
• Domain Diversity: The dataset covers 12 professional domains, including Healthcare, Finance, Manufacturing, and various STEM engineering fields. Chart types include niche formats such as Moody diagrams, Bode plots, wind roses, and pediatric growth charts.
• Authoring Process:
  • Expert Authoring: Tasks are authored by domain experts who provide the intended answer, a complete walkthrough of visual and computational steps, and an acceptable range for visually estimated quantities.
  • Independent Verification: Each task is independently verified by three additional domain experts to validate the reading, clarify wording, and confirm acceptable ranges.
  • Adversarial Screening: Tasks are screened against frontier models; a task is retained only if it elicits a meaningful failure in at least one frontier model.
• Grading Protocol:
  • Expert-Calibrated Ranges: Unlike benchmarks using blanket tolerances, Chartography uses ranges set by experts based on the chart's resolution and gridline density (e.g., ±0.5 cm for a growth chart vs. 140–150 mph for a wind map).
  • All-or-Nothing Scoring: Multi-part answers are graded strictly; missing a single component results in failure.
  • Judge Mechanism: A model judge (Gemini 3.5 Flash) evaluates responses against the golden answer and acceptable ranges without seeing the chart image, ensuring it adjudicates answer equivalence rather than re-solving the visual task.

Evaluation Protocol

• Models Evaluated: 30 configurations of 18 frontier models from eight developers (including OpenAI, Google, Anthropic, Meta, xAI, etc.).
• Constraints: Models receive a single user message with the prompt and image. No system prompts, few-shot examples, tools (code execution, zooming), or image normalization are permitted.
• Metrics: Each configuration undergoes 20 trials per task (2,000 total trials). The primary metric is mean pass@1 (the fraction of trials where the model's final answer falls within the acceptable range).

Key Contributions

1. Task Definition and Dataset: A definition of professional chart understanding as answering questions using only the chart and standing domain knowledge. The release includes 100 expert-authored tasks, images, provenance metadata, and evaluation code.
2. Expert-Calibrated Grading: A protocol replacing blanket numeric tolerances with expert-set acceptable ranges and all-or-nothing grading for multi-part answers.
3. Comprehensive Evaluation: An evaluation of 30 frontier model configurations revealing a significant performance gap compared to other benchmarks.
4. Audited Release: Full transparency with composition and provenance metadata to allow independent verification of results.

Results

• Performance Ceiling: The best-performing configuration (GPT-5.6 Sol with maximum reasoning effort) achieved a mean pass@1 of 45.0%. The remaining configurations scored between 9.0% and 39.5%. This stands in stark contrast to the 80–90% scores these models achieve on benchmarks like ChartQA and CharXiv.
• Reasoning Effort: Increasing reasoning effort (e.g., "max" or "adaptive" modes) improved performance in 11 of 12 matched model pairs, with a median uplift of +4.5 percentage points. However, the gains were inconsistent, and in some cases (e.g., Grok 4.5), higher reasoning effort led to lower scores.
• Cost vs. Accuracy: The Pareto frontier for cost versus accuracy is steep at the low end but flattens significantly; moving from ~40% to 45% accuracy requires a several-fold increase in inference cost. Some high-reasoning configurations generated 20,000–30,000+ tokens per trial without commensurate accuracy gains.

Failure Modes

Analysis of failures identified four primary categories:

1. Missed Visual Elements: Models fail to detect thin, faint, or overlapping features (e.g., missing smaller flows in a Sankey diagram).
2. Correct Feature, Incorrect Value: Models identify the correct curve or contour but misjudge its position (e.g., reading a peak two gridlines off or misinterpreting a non-linear scale).
3. Complex Geometry: Significant struggles with 3D surface plots and projected geometry, where models fail to use projected gridlines to interpret depth and coordinates.
4. Domain Conventions: Models fail to apply implicit professional rules encoded in the chart (e.g., using the lowest plotted value for a curve when the query is below the range, rather than extrapolating).

Significance and Claims

The paper posits that Chartography reveals a critical gap in current multimodal AI capabilities: visual grounding. While models demonstrate strong semantic recognition (identifying what a chart represents), they struggle with metric grounding (anchoring numeric answers to specific visual positions).

The authors claim that:

• Benchmark Saturation: High performance on existing academic benchmarks does not indicate readiness for professional deployment in clinical, engineering, or financial contexts.
• Bottleneck Identification: The primary limitation for frontier models in this domain is visual perception, not reasoning depth. Additional deliberation often elaborates on incorrect visual observations rather than correcting them.
• Prerequisites for Deployment: Accurate chart reading, including adherence to domain conventions and calibrated visual estimation, is a prerequisite for proposed applications like clinical decision support and engineering review.
• Future Direction: Improving visual reading capabilities, holding reasoning fixed, is likely to yield substantial performance gains. The benchmark serves as a tool to measure progress toward these realistic professional use cases.