InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information

1. Problem Statement

Current Vision-Language Models (VLMs) have demonstrated strong capabilities in answering questions about single, isolated charts (Visual Question Answering or VQA). However, real-world data analysis (e.g., in scientific reports, financial dashboards, and policy documents) rarely relies on a single visualization. Instead, insights are derived by synthesizing information across multiple, heterogeneous charts that may differ in style, domain, temporal alignment, and semantic framing.

Existing benchmarks for multi-chart reasoning suffer from several limitations:

• Simplification: They often rely on synthetic data with uniform styles and static layouts.
• Lack of Complexity: They fail to capture real-world challenges such as visual inconsistency, semantic misalignment, temporal discontinuity, and the need for multi-step aggregation.
• Evaluation Flaws: They typically use exact string matching, which fails to account for paraphrased answers, numerical approximations, or equivalent units, leading to an inaccurate assessment of semantic understanding.

INTERCHART addresses these gaps by introducing a diagnostic benchmark designed to evaluate how well VLMs reason across multiple related charts with increasing levels of complexity, visual diversity, and semantic drift.

2. Methodology

The INTERCHART benchmark is structured into three distinct tiers (subsets), each designed to isolate specific reasoning challenges rather than establish a simple performance hierarchy. The dataset contains 5,214 validated question-answer (QA) pairs across 1,012 multi-chart contexts.

A. Benchmark Subsets

1. DECAF (Decomposed Elementary Charts with Answerable Facts):
   • Goal: Establish a baseline for atomic fact retrieval and localized comparisons.
   • Construction: Takes complex, multi-variable charts from existing datasets (ChartQA, DVQA, etc.) and decomposes them into simplified, single-variable charts using a custom pipeline (DePlot for table extraction, Plotly for rendering).
   • Focus: Factual lookup and parallel reasoning within visually simplified units.
2. SPECTRA (Synthetic Plots for Event-based Correlated Trend Reasoning and Analysis):
   • Goal: Evaluate the ability to integrate distributed information across visually distinct but thematically aligned synthetic charts.
   • Construction: Uses LLMs to generate structured tables with shared axes (e.g., GDP vs. Life Expectancy) and renders them into diverse chart types (bar-line overlays, multi-axis).
   • Focus: Correlated trend reasoning, event-based inference, and aligning related quantities across chart pairs.
3. STORM (Sequential Temporal Reasoning Over Real-world Multi-domain charts):
   • Goal: Probe the upper limits of VLM capabilities with complex, real-world data.
   • Construction: Curated from "Our World in Data," featuring real-world line chart pairs with diverse styles, domains, and temporal misalignments (e.g., economic reports, public health dashboards).
   • Focus: Semantic abstraction, temporal alignment, and multi-step integration across heterogeneous visual styles.

B. Evaluation Pipeline

To overcome the limitations of string matching, the authors propose a novel LLM-assisted evaluation framework:

• Semantic Judges: Three distinct LLMs (Gemini 1.5 Flash, Phi-4, and Qwen2.5-7B) act as judges.
• Majority Voting: Each model evaluates the VLM's output against the ground truth. A binary correctness score is assigned based on majority voting.
• Flexibility: This approach accepts paraphrased answers, numeric approximations, and equivalent units, providing a more robust measure of semantic understanding.

C. Experimental Setup

The authors evaluated a diverse set of closed-source (Gemini 1.5 Pro, GPT-4o Mini) and open-source (Qwen2-VL, MiniCPM-V, InternVL, Idefics3) VLMs. They tested multiple input strategies:

• Visual Formats: Combined (stitched images) vs. Interleaved (sequential inputs).
• Prompting: Zero-Shot, Zero-Shot Chain-of-Thought (CoT), and Few-Shot with Directives.
• Intermediate Representations: Direct visual input vs. Chart-to-Table conversion (using DePlot and DePlot++).

3. Key Contributions

1. First Multi-Tier Benchmark: INTERCHART is the first benchmark to span decomposed, synthetic, and real-world chart contexts, specifically targeting multi-chart reasoning.
2. Structured Reasoning Tasks: It introduces a tiered system (DECAF, SPECTRA, STORM) to disentangle visual parsing errors from reasoning failures, capturing localized dependencies, trend correlations, and temporal abstractions.
3. Semantic Evaluation Framework: The LLM-based majority voting system improves alignment with human judgment and enables fine-grained error analysis, moving beyond rigid string matching.
4. Diagnostic Insights: The benchmark reveals that current models struggle significantly with cross-chart integration, particularly when visual styles and semantic contexts diverge.

4. Key Results and Analysis

The evaluation of state-of-the-art VLMs revealed consistent and steep accuracy declines as chart complexity increased:

• Performance Drop: Models performed well on DECAF (simplified charts) but saw significant accuracy drops in SPECTRA and a drastic decline in STORM. For example, Gemini-1.5 Pro dropped from ~65% in DECAF to ~35% in STORM.
• Decomposition Helps: Converting complex charts into decomposed, single-variable units (DECAF) significantly improved accuracy, highlighting that models struggle with cross-chart integration and visual clutter rather than basic fact retrieval.
• Synthetic vs. Real-World: Models generalize poorly from synthetic data (SPECTRA) to real-world scenarios (STORM). The shift involves handling domain shifts, visual diversity, and temporal reasoning, which current models fail to transfer effectively.
• Prompting Strategies: Chain-of-Thought (CoT) and Few-Shot prompting with directives improved performance on simpler subsets (DECAF/SPECTRA) but offered limited benefit for the highly complex STORM subset, suggesting verbal reasoning cannot compensate for high visual/semantic complexity.
• Intermediate Representations: While converting charts to tables (Chart-to-Table) helped on synthetic data, it degraded performance on real-world STORM charts. This indicates that tables often fail to capture the semantic and temporal alignment critical for reasoning across misaligned real-world axes.
• Model Comparison: Gemini-1.5 Pro consistently outperformed other models across all tiers, likely due to its training on structured inputs and strong instruction-following. Open-source models (Qwen2, MiniCPM) showed reasonable performance on simple tasks but struggled with complex reasoning.

5. Significance

INTERCHART provides a rigorous diagnostic framework that shifts the focus from "can the model answer?" to "where and why does the model fail?"

• Identifies Failure Modes: It isolates specific weaknesses, such as the inability to handle semantic drift, temporal misalignment, and visual heterogeneity.
• Guides Future Research: The results suggest that future VLMs need better mechanisms for cross-visual attention, temporal reasoning, and semantic alignment rather than just improved visual parsing.
• Real-World Applicability: By focusing on real-world multi-chart scenarios (STORM), the benchmark directly addresses the needs of applications in scientific reporting, financial analysis, and public policy, where data synthesis is critical.

In conclusion, INTERCHART demonstrates that while VLMs are becoming proficient at reading individual charts, they remain fundamentally challenged by the task of synthesizing information across multiple, diverse visualizations—a critical gap for real-world AI deployment.